2023-03-31 18:53:30,555 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82 2023-03-31 18:53:30,570 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-31 18:53:30,590 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-31 18:53:30,590 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/cluster_a78361e5-c460-aaf7-5d30-b8a2ca8e7b70, deleteOnExit=true 2023-03-31 18:53:30,591 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-31 18:53:30,591 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/test.cache.data in system properties and HBase conf 2023-03-31 18:53:30,592 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/hadoop.tmp.dir in system properties and HBase conf 2023-03-31 18:53:30,592 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/hadoop.log.dir in system properties and HBase conf 2023-03-31 18:53:30,592 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-31 18:53:30,593 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-31 18:53:30,593 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-31 18:53:30,708 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-31 18:53:31,141 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-31 18:53:31,147 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-31 18:53:31,148 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-31 18:53:31,149 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-31 18:53:31,149 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-31 18:53:31,150 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-31 18:53:31,150 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-31 18:53:31,151 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-31 18:53:31,153 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-31 18:53:31,154 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-31 18:53:31,154 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/nfs.dump.dir in system properties and HBase conf 2023-03-31 18:53:31,154 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/java.io.tmpdir in system properties and HBase conf 2023-03-31 18:53:31,155 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-31 18:53:31,155 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-31 18:53:31,155 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-31 18:53:31,722 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-31 18:53:31,727 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-31 18:53:32,048 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-31 18:53:32,240 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-31 18:53:32,254 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-31 18:53:32,291 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-31 18:53:32,323 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/java.io.tmpdir/Jetty_localhost_46419_hdfs____ld72v8/webapp 2023-03-31 18:53:32,469 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46419 2023-03-31 18:53:32,521 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-31 18:53:32,521 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-31 18:53:32,957 WARN [Listener at localhost/44569] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-31 18:53:33,093 WARN [Listener at localhost/44569] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-31 18:53:33,112 WARN [Listener at localhost/44569] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-31 18:53:33,120 INFO [Listener at localhost/44569] log.Slf4jLog(67): jetty-6.1.26 2023-03-31 18:53:33,125 INFO [Listener at localhost/44569] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/java.io.tmpdir/Jetty_localhost_45305_datanode____lghc5a/webapp 2023-03-31 18:53:33,224 INFO [Listener at localhost/44569] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45305 2023-03-31 18:53:33,600 WARN [Listener at localhost/43327] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-31 18:53:34,190 DEBUG [Listener at localhost/43327] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82 2023-03-31 18:53:34,197 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x135ea6b405b2c40e: Processing first storage report for DS-f00272e4-d22a-446d-af98-f1278fc1eb58 from datanode 2ae9146c-fb95-48c5-9b36-7f399d9cd4f0 2023-03-31 18:53:34,199 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x135ea6b405b2c40e: from storage DS-f00272e4-d22a-446d-af98-f1278fc1eb58 node DatanodeRegistration(127.0.0.1:40795, datanodeUuid=2ae9146c-fb95-48c5-9b36-7f399d9cd4f0, infoPort=36541, infoSecurePort=0, ipcPort=43327, storageInfo=lv=-57;cid=testClusterID;nsid=1145832913;c=1680288811800), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-31 18:53:34,199 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x135ea6b405b2c40e: Processing first storage report for DS-5cf5dbe1-4a7f-4124-9725-7927d27c3257 from datanode 2ae9146c-fb95-48c5-9b36-7f399d9cd4f0 2023-03-31 18:53:34,200 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x135ea6b405b2c40e: from storage DS-5cf5dbe1-4a7f-4124-9725-7927d27c3257 node DatanodeRegistration(127.0.0.1:40795, datanodeUuid=2ae9146c-fb95-48c5-9b36-7f399d9cd4f0, infoPort=36541, infoSecurePort=0, ipcPort=43327, storageInfo=lv=-57;cid=testClusterID;nsid=1145832913;c=1680288811800), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-31 18:53:34,268 INFO [Listener at localhost/43327] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/cluster_a78361e5-c460-aaf7-5d30-b8a2ca8e7b70/zookeeper_0, clientPort=63858, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/cluster_a78361e5-c460-aaf7-5d30-b8a2ca8e7b70/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/cluster_a78361e5-c460-aaf7-5d30-b8a2ca8e7b70/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-31 18:53:34,285 INFO [Listener at localhost/43327] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63858 2023-03-31 18:53:34,293 INFO [Listener at localhost/43327] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 18:53:34,296 INFO [Listener at localhost/43327] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 18:53:34,951 INFO [Listener at localhost/43327] util.FSUtils(471): Created version file at hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9 with version=8 2023-03-31 18:53:34,951 INFO [Listener at localhost/43327] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/hbase-staging 2023-03-31 18:53:35,279 INFO [Listener at localhost/43327] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-31 18:53:35,790 INFO [Listener at localhost/43327] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-31 18:53:35,821 INFO [Listener at localhost/43327] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-31 18:53:35,822 INFO [Listener at localhost/43327] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-31 18:53:35,822 INFO [Listener at localhost/43327] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-31 18:53:35,822 INFO [Listener at localhost/43327] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-31 18:53:35,822 INFO [Listener at localhost/43327] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-31 18:53:35,967 INFO [Listener at localhost/43327] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-31 18:53:36,042 DEBUG [Listener at localhost/43327] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-31 18:53:36,137 INFO [Listener at localhost/43327] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35737 2023-03-31 18:53:36,146 INFO [Listener at localhost/43327] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 18:53:36,149 INFO [Listener at localhost/43327] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 18:53:36,169 INFO [Listener at localhost/43327] zookeeper.RecoverableZooKeeper(93): Process identifier=master:35737 connecting to ZooKeeper ensemble=127.0.0.1:63858 2023-03-31 18:53:36,210 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:357370x0, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-31 18:53:36,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:35737-0x1013bec7e730000 connected 2023-03-31 18:53:36,243 DEBUG [Listener at localhost/43327] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-31 18:53:36,244 DEBUG [Listener at localhost/43327] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-31 18:53:36,248 DEBUG [Listener at localhost/43327] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-31 18:53:36,256 DEBUG [Listener at localhost/43327] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35737 2023-03-31 18:53:36,257 DEBUG [Listener at localhost/43327] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35737 2023-03-31 18:53:36,257 DEBUG [Listener at localhost/43327] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35737 2023-03-31 18:53:36,258 DEBUG [Listener at localhost/43327] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35737 2023-03-31 18:53:36,258 DEBUG [Listener at localhost/43327] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35737 2023-03-31 18:53:36,264 INFO [Listener at localhost/43327] master.HMaster(444): hbase.rootdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9, hbase.cluster.distributed=false 2023-03-31 18:53:36,340 INFO [Listener at localhost/43327] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-31 18:53:36,340 INFO [Listener at localhost/43327] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-31 18:53:36,341 INFO [Listener at localhost/43327] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-31 18:53:36,341 INFO [Listener at localhost/43327] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-31 18:53:36,341 INFO [Listener at localhost/43327] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-31 18:53:36,341 INFO [Listener at localhost/43327] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-31 18:53:36,346 INFO [Listener at localhost/43327] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-31 18:53:36,349 INFO [Listener at localhost/43327] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34951 2023-03-31 18:53:36,351 INFO [Listener at localhost/43327] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-31 18:53:36,357 DEBUG [Listener at localhost/43327] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-31 18:53:36,357 INFO [Listener at localhost/43327] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 18:53:36,360 INFO [Listener at localhost/43327] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 18:53:36,362 INFO [Listener at localhost/43327] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34951 connecting to ZooKeeper ensemble=127.0.0.1:63858 2023-03-31 18:53:36,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:349510x0, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-31 18:53:36,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34951-0x1013bec7e730001 connected 2023-03-31 18:53:36,367 DEBUG [Listener at localhost/43327] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-31 18:53:36,368 DEBUG [Listener at localhost/43327] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-31 18:53:36,369 DEBUG [Listener at localhost/43327] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-31 18:53:36,370 DEBUG [Listener at localhost/43327] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34951 2023-03-31 18:53:36,370 DEBUG [Listener at localhost/43327] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34951 2023-03-31 18:53:36,370 DEBUG [Listener at localhost/43327] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34951 2023-03-31 18:53:36,371 DEBUG [Listener at localhost/43327] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34951 2023-03-31 18:53:36,371 DEBUG [Listener at localhost/43327] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34951 2023-03-31 18:53:36,375 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,35737,1680288815092 2023-03-31 18:53:36,384 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-31 18:53:36,386 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,35737,1680288815092 2023-03-31 18:53:36,405 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-31 18:53:36,405 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-31 18:53:36,405 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 18:53:36,406 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-31 18:53:36,407 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,35737,1680288815092 from backup master directory 2023-03-31 18:53:36,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-31 18:53:36,410 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,35737,1680288815092 2023-03-31 18:53:36,410 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-31 18:53:36,410 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-31 18:53:36,411 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,35737,1680288815092 2023-03-31 18:53:36,413 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-31 18:53:36,415 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-31 18:53:36,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/hbase.id with ID: 9b725434-741e-480d-ba7a-7ae736c99373 2023-03-31 18:53:36,535 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 18:53:36,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 18:53:36,593 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6ffc8fa3 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:36,624 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7791587d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:36,648 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 18:53:36,650 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-31 18:53:36,669 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-31 18:53:36,669 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-31 18:53:36,672 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:53:36,676 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:53:36,677 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-31 18:53:36,709 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/data/master/store-tmp 2023-03-31 18:53:36,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:53:36,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-31 18:53:36,736 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 18:53:36,736 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 18:53:36,736 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-31 18:53:36,736 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 18:53:36,736 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 18:53:36,736 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-31 18:53:36,737 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/WALs/jenkins-hbase4.apache.org,35737,1680288815092 2023-03-31 18:53:36,764 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35737%2C1680288815092, suffix=, logDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/WALs/jenkins-hbase4.apache.org,35737,1680288815092, archiveDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/oldWALs, maxLogs=10 2023-03-31 18:53:36,814 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40795,DS-f00272e4-d22a-446d-af98-f1278fc1eb58,DISK] 2023-03-31 18:53:36,822 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:53:36,889 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/WALs/jenkins-hbase4.apache.org,35737,1680288815092/jenkins-hbase4.apache.org%2C35737%2C1680288815092.1680288816773 2023-03-31 18:53:36,889 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40795,DS-f00272e4-d22a-446d-af98-f1278fc1eb58,DISK]] 2023-03-31 18:53:36,890 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:53:36,890 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:53:36,893 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-31 18:53:36,894 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-31 18:53:36,957 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-31 18:53:36,964 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-31 18:53:36,988 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-31 18:53:37,000 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:37,006 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-31 18:53:37,007 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-31 18:53:37,022 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-31 18:53:37,026 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 18:53:37,028 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61353813, jitterRate=-0.08575694262981415}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-31 18:53:37,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-31 18:53:37,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-31 18:53:37,051 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-31 18:53:37,051 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-31 18:53:37,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-31 18:53:37,056 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-31 18:53:37,088 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-03-31 18:53:37,088 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-31 18:53:37,117 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-31 18:53:37,122 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-31 18:53:37,149 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-31 18:53:37,154 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-31 18:53:37,156 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-31 18:53:37,162 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-31 18:53:37,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-31 18:53:37,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 18:53:37,176 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-31 18:53:37,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-31 18:53:37,189 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-31 18:53:37,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-31 18:53:37,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-31 18:53:37,194 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 18:53:37,195 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,35737,1680288815092, sessionid=0x1013bec7e730000, setting cluster-up flag (Was=false) 2023-03-31 18:53:37,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 18:53:37,220 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-31 18:53:37,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35737,1680288815092 2023-03-31 18:53:37,228 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 18:53:37,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-31 18:53:37,236 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35737,1680288815092 2023-03-31 18:53:37,240 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.hbase-snapshot/.tmp 2023-03-31 18:53:37,277 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(952): ClusterId : 9b725434-741e-480d-ba7a-7ae736c99373 2023-03-31 18:53:37,282 DEBUG [RS:0;jenkins-hbase4:34951] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-31 18:53:37,287 DEBUG [RS:0;jenkins-hbase4:34951] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-31 18:53:37,287 DEBUG [RS:0;jenkins-hbase4:34951] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-31 18:53:37,295 DEBUG [RS:0;jenkins-hbase4:34951] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-31 18:53:37,296 DEBUG [RS:0;jenkins-hbase4:34951] zookeeper.ReadOnlyZKClient(139): Connect 0x3b3fb4aa to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:37,305 DEBUG [RS:0;jenkins-hbase4:34951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35f85dbd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:37,305 DEBUG [RS:0;jenkins-hbase4:34951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31c7dfda, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-31 18:53:37,338 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34951 2023-03-31 18:53:37,343 INFO [RS:0;jenkins-hbase4:34951] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-31 18:53:37,343 INFO [RS:0;jenkins-hbase4:34951] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-31 18:53:37,343 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1023): About to register with Master. 2023-03-31 18:53:37,347 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35737,1680288815092 with isa=jenkins-hbase4.apache.org/172.31.14.131:34951, startcode=1680288816339 2023-03-31 18:53:37,369 DEBUG [RS:0;jenkins-hbase4:34951] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-31 18:53:37,386 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-31 18:53:37,400 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-31 18:53:37,400 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-31 18:53:37,400 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-31 18:53:37,400 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-31 18:53:37,401 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-31 18:53:37,401 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,401 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-31 18:53:37,401 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,402 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680288847402 2023-03-31 18:53:37,405 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-31 18:53:37,409 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-31 18:53:37,410 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-31 18:53:37,421 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-31 18:53:37,422 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-31 18:53:37,430 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37193, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-31 18:53:37,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-31 18:53:37,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-31 18:53:37,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-31 18:53:37,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-31 18:53:37,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:37,445 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-31 18:53:37,447 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-31 18:53:37,447 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-31 18:53:37,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:37,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-31 18:53:37,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-31 18:53:37,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680288817456,5,FailOnTimeoutGroup] 2023-03-31 18:53:37,459 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680288817456,5,FailOnTimeoutGroup] 2023-03-31 18:53:37,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:37,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-31 18:53:37,460 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:37,461 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:37,487 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-31 18:53:37,488 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-31 18:53:37,489 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9 2023-03-31 18:53:37,490 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(2829): Master is not running yet 2023-03-31 18:53:37,490 WARN [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-03-31 18:53:37,502 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:53:37,504 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-31 18:53:37,507 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/info 2023-03-31 18:53:37,508 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-31 18:53:37,509 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:37,510 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-31 18:53:37,513 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/rep_barrier 2023-03-31 18:53:37,514 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-31 18:53:37,514 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:37,515 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-31 18:53:37,517 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/table 2023-03-31 18:53:37,518 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-31 18:53:37,519 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:37,521 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740 2023-03-31 18:53:37,522 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740 2023-03-31 18:53:37,526 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:53:37,528 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-31 18:53:37,532 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 18:53:37,533 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74713121, jitterRate=0.1133122593164444}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:53:37,533 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-31 18:53:37,533 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-31 18:53:37,533 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-31 18:53:37,534 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-31 18:53:37,534 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-31 18:53:37,534 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-31 18:53:37,534 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-31 18:53:37,535 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-31 18:53:37,540 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-31 18:53:37,540 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-31 18:53:37,548 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-31 18:53:37,560 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-31 18:53:37,562 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-31 18:53:37,591 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35737,1680288815092 with isa=jenkins-hbase4.apache.org/172.31.14.131:34951, startcode=1680288816339 2023-03-31 18:53:37,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:37,604 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9 2023-03-31 18:53:37,605 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44569 2023-03-31 18:53:37,605 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-31 18:53:37,612 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-31 18:53:37,614 DEBUG [RS:0;jenkins-hbase4:34951] zookeeper.ZKUtil(162): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:37,614 WARN [RS:0;jenkins-hbase4:34951] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-31 18:53:37,614 INFO [RS:0;jenkins-hbase4:34951] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-31 18:53:37,615 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/WALs/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:37,615 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34951,1680288816339] 2023-03-31 18:53:37,625 DEBUG [RS:0;jenkins-hbase4:34951] zookeeper.ZKUtil(162): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:37,636 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-31 18:53:37,644 INFO [RS:0;jenkins-hbase4:34951] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-31 18:53:37,665 INFO [RS:0;jenkins-hbase4:34951] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-31 18:53:37,669 INFO [RS:0;jenkins-hbase4:34951] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-31 18:53:37,669 INFO [RS:0;jenkins-hbase4:34951] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:37,672 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-31 18:53:37,679 INFO [RS:0;jenkins-hbase4:34951] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:37,679 DEBUG [RS:0;jenkins-hbase4:34951] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,680 DEBUG [RS:0;jenkins-hbase4:34951] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,680 DEBUG [RS:0;jenkins-hbase4:34951] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,680 DEBUG [RS:0;jenkins-hbase4:34951] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,680 DEBUG [RS:0;jenkins-hbase4:34951] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,680 DEBUG [RS:0;jenkins-hbase4:34951] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-31 18:53:37,680 DEBUG [RS:0;jenkins-hbase4:34951] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,680 DEBUG [RS:0;jenkins-hbase4:34951] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,680 DEBUG [RS:0;jenkins-hbase4:34951] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,681 DEBUG [RS:0;jenkins-hbase4:34951] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 18:53:37,681 INFO [RS:0;jenkins-hbase4:34951] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:37,681 INFO [RS:0;jenkins-hbase4:34951] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:37,681 INFO [RS:0;jenkins-hbase4:34951] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:37,697 INFO [RS:0;jenkins-hbase4:34951] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-31 18:53:37,700 INFO [RS:0;jenkins-hbase4:34951] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34951,1680288816339-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:37,715 DEBUG [jenkins-hbase4:35737] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-31 18:53:37,717 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34951,1680288816339, state=OPENING 2023-03-31 18:53:37,720 INFO [RS:0;jenkins-hbase4:34951] regionserver.Replication(203): jenkins-hbase4.apache.org,34951,1680288816339 started 2023-03-31 18:53:37,720 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34951,1680288816339, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34951, sessionid=0x1013bec7e730001 2023-03-31 18:53:37,720 DEBUG [RS:0;jenkins-hbase4:34951] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-31 18:53:37,720 DEBUG [RS:0;jenkins-hbase4:34951] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:37,721 DEBUG [RS:0;jenkins-hbase4:34951] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34951,1680288816339' 2023-03-31 18:53:37,721 DEBUG [RS:0;jenkins-hbase4:34951] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:37,721 DEBUG [RS:0;jenkins-hbase4:34951] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:37,722 DEBUG [RS:0;jenkins-hbase4:34951] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-31 18:53:37,722 DEBUG [RS:0;jenkins-hbase4:34951] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-31 18:53:37,722 DEBUG [RS:0;jenkins-hbase4:34951] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:37,722 DEBUG [RS:0;jenkins-hbase4:34951] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34951,1680288816339' 2023-03-31 18:53:37,722 DEBUG [RS:0;jenkins-hbase4:34951] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-31 18:53:37,723 DEBUG [RS:0;jenkins-hbase4:34951] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-31 18:53:37,724 DEBUG [RS:0;jenkins-hbase4:34951] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-31 18:53:37,724 INFO [RS:0;jenkins-hbase4:34951] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-31 18:53:37,724 INFO [RS:0;jenkins-hbase4:34951] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-31 18:53:37,725 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-31 18:53:37,727 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 18:53:37,728 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-31 18:53:37,731 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:53:37,836 INFO [RS:0;jenkins-hbase4:34951] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34951%2C1680288816339, suffix=, logDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/WALs/jenkins-hbase4.apache.org,34951,1680288816339, archiveDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/oldWALs, maxLogs=32 2023-03-31 18:53:37,860 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40795,DS-f00272e4-d22a-446d-af98-f1278fc1eb58,DISK] 2023-03-31 18:53:37,865 INFO [RS:0;jenkins-hbase4:34951] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/WALs/jenkins-hbase4.apache.org,34951,1680288816339/jenkins-hbase4.apache.org%2C34951%2C1680288816339.1680288817840 2023-03-31 18:53:37,865 DEBUG [RS:0;jenkins-hbase4:34951] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40795,DS-f00272e4-d22a-446d-af98-f1278fc1eb58,DISK]] 2023-03-31 18:53:37,914 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:37,916 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-31 18:53:37,920 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59962, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-31 18:53:37,930 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-31 18:53:37,931 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-31 18:53:37,934 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34951%2C1680288816339.meta, suffix=.meta, logDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/WALs/jenkins-hbase4.apache.org,34951,1680288816339, archiveDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/oldWALs, maxLogs=32 2023-03-31 18:53:37,951 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40795,DS-f00272e4-d22a-446d-af98-f1278fc1eb58,DISK] 2023-03-31 18:53:37,955 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/WALs/jenkins-hbase4.apache.org,34951,1680288816339/jenkins-hbase4.apache.org%2C34951%2C1680288816339.meta.1680288817935.meta 2023-03-31 18:53:37,955 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40795,DS-f00272e4-d22a-446d-af98-f1278fc1eb58,DISK]] 2023-03-31 18:53:37,956 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:53:37,957 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-31 18:53:37,974 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-31 18:53:37,979 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-31 18:53:37,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-31 18:53:37,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:53:37,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-31 18:53:37,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-31 18:53:37,988 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-31 18:53:37,990 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/info 2023-03-31 18:53:37,990 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/info 2023-03-31 18:53:37,990 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-31 18:53:37,991 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:37,991 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-31 18:53:37,993 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/rep_barrier 2023-03-31 18:53:37,993 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/rep_barrier 2023-03-31 18:53:37,993 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-31 18:53:37,994 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:37,994 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-31 18:53:37,996 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/table 2023-03-31 18:53:37,996 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/table 2023-03-31 18:53:37,996 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-31 18:53:37,997 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:37,999 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740 2023-03-31 18:53:38,001 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740 2023-03-31 18:53:38,005 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:53:38,007 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-31 18:53:38,008 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63487884, jitterRate=-0.053956806659698486}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:53:38,009 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-31 18:53:38,020 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680288817906 2023-03-31 18:53:38,037 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-31 18:53:38,038 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-31 18:53:38,038 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34951,1680288816339, state=OPEN 2023-03-31 18:53:38,041 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-31 18:53:38,041 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-31 18:53:38,046 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-31 18:53:38,046 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34951,1680288816339 in 311 msec 2023-03-31 18:53:38,051 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-31 18:53:38,051 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 499 msec 2023-03-31 18:53:38,056 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 749 msec 2023-03-31 18:53:38,057 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680288818056, completionTime=-1 2023-03-31 18:53:38,057 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-31 18:53:38,057 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-31 18:53:38,123 DEBUG [hconnection-0x787d1ab5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:38,127 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59968, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:38,143 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-31 18:53:38,143 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680288878143 2023-03-31 18:53:38,144 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680288938144 2023-03-31 18:53:38,144 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 86 msec 2023-03-31 18:53:38,166 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35737,1680288815092-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:38,167 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35737,1680288815092-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:38,167 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35737,1680288815092-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:38,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:35737, period=300000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:38,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-31 18:53:38,174 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-31 18:53:38,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-31 18:53:38,186 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-31 18:53:38,195 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-31 18:53:38,197 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 18:53:38,200 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 18:53:38,222 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:53:38,224 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536 empty. 2023-03-31 18:53:38,225 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:53:38,225 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-31 18:53:38,646 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-31 18:53:38,648 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2599f97ef8c905b3cf5450aad14b5536, NAME => 'hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp 2023-03-31 18:53:38,660 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:53:38,660 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 2599f97ef8c905b3cf5450aad14b5536, disabling compactions & flushes 2023-03-31 18:53:38,660 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:53:38,660 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:53:38,660 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. after waiting 0 ms 2023-03-31 18:53:38,660 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:53:38,661 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:53:38,661 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 2599f97ef8c905b3cf5450aad14b5536: 2023-03-31 18:53:38,665 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 18:53:38,679 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680288818668"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288818668"}]},"ts":"1680288818668"} 2023-03-31 18:53:38,705 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 18:53:38,708 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 18:53:38,712 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288818708"}]},"ts":"1680288818708"} 2023-03-31 18:53:38,716 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-31 18:53:38,731 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=2599f97ef8c905b3cf5450aad14b5536, ASSIGN}] 2023-03-31 18:53:38,734 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=2599f97ef8c905b3cf5450aad14b5536, ASSIGN 2023-03-31 18:53:38,736 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=2599f97ef8c905b3cf5450aad14b5536, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34951,1680288816339; forceNewPlan=false, retain=false 2023-03-31 18:53:38,887 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=2599f97ef8c905b3cf5450aad14b5536, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:38,887 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680288818887"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288818887"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288818887"}]},"ts":"1680288818887"} 2023-03-31 18:53:38,892 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 2599f97ef8c905b3cf5450aad14b5536, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:53:39,052 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:53:39,053 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2599f97ef8c905b3cf5450aad14b5536, NAME => 'hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:53:39,054 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:53:39,054 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:53:39,054 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:53:39,054 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:53:39,056 INFO [StoreOpener-2599f97ef8c905b3cf5450aad14b5536-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:53:39,059 DEBUG [StoreOpener-2599f97ef8c905b3cf5450aad14b5536-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536/info 2023-03-31 18:53:39,059 DEBUG [StoreOpener-2599f97ef8c905b3cf5450aad14b5536-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536/info 2023-03-31 18:53:39,059 INFO [StoreOpener-2599f97ef8c905b3cf5450aad14b5536-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2599f97ef8c905b3cf5450aad14b5536 columnFamilyName info 2023-03-31 18:53:39,060 INFO [StoreOpener-2599f97ef8c905b3cf5450aad14b5536-1] regionserver.HStore(310): Store=2599f97ef8c905b3cf5450aad14b5536/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:39,062 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:53:39,062 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:53:39,067 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:53:39,070 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 18:53:39,071 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2599f97ef8c905b3cf5450aad14b5536; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67197153, jitterRate=0.0013156086206436157}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-31 18:53:39,071 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2599f97ef8c905b3cf5450aad14b5536: 2023-03-31 18:53:39,074 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536., pid=6, masterSystemTime=1680288819045 2023-03-31 18:53:39,077 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:53:39,077 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:53:39,079 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=2599f97ef8c905b3cf5450aad14b5536, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:39,079 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680288819078"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680288819078"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680288819078"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680288819078"}]},"ts":"1680288819078"} 2023-03-31 18:53:39,087 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-31 18:53:39,087 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 2599f97ef8c905b3cf5450aad14b5536, server=jenkins-hbase4.apache.org,34951,1680288816339 in 191 msec 2023-03-31 18:53:39,090 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-31 18:53:39,090 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=2599f97ef8c905b3cf5450aad14b5536, ASSIGN in 356 msec 2023-03-31 18:53:39,091 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 18:53:39,091 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288819091"}]},"ts":"1680288819091"} 2023-03-31 18:53:39,094 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-31 18:53:39,097 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 18:53:39,098 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-31 18:53:39,100 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 911 msec 2023-03-31 18:53:39,100 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-31 18:53:39,100 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 18:53:39,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-31 18:53:39,160 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-31 18:53:39,166 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 40 msec 2023-03-31 18:53:39,174 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-31 18:53:39,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-31 18:53:39,193 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-03-31 18:53:39,201 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-31 18:53:39,205 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-31 18:53:39,206 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.795sec 2023-03-31 18:53:39,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-31 18:53:39,210 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-31 18:53:39,210 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-31 18:53:39,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35737,1680288815092-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-31 18:53:39,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35737,1680288815092-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-31 18:53:39,223 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-31 18:53:39,286 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x5e324982 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:39,292 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@373f9c99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:39,308 DEBUG [hconnection-0x258148f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:39,319 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59974, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:39,330 INFO [Listener at localhost/43327] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,35737,1680288815092 2023-03-31 18:53:39,354 INFO [Listener at localhost/43327] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=312, ProcessCount=169, AvailableMemoryMB=4564 2023-03-31 18:53:39,371 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 18:53:39,375 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60658, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 18:53:39,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 18:53:39,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 18:53:39,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 18:53:39,396 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 18:53:39,398 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 18:53:39,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-31 18:53:39,402 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:39,403 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134 empty. 2023-03-31 18:53:39,405 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:39,405 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:53:39,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-31 18:53:39,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-31 18:53:39,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-31 18:53:39,823 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 18:53:39,825 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 087bf54664c6a240c659c71fb3782134, NAME => 'TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp 2023-03-31 18:53:39,836 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:53:39,836 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 087bf54664c6a240c659c71fb3782134, disabling compactions & flushes 2023-03-31 18:53:39,836 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:39,836 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:39,836 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. after waiting 0 ms 2023-03-31 18:53:39,836 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:39,837 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:39,837 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:39,841 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 18:53:39,842 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288819842"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288819842"}]},"ts":"1680288819842"} 2023-03-31 18:53:39,845 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 18:53:39,846 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 18:53:39,846 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288819846"}]},"ts":"1680288819846"} 2023-03-31 18:53:39,849 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 18:53:39,852 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=087bf54664c6a240c659c71fb3782134, ASSIGN}] 2023-03-31 18:53:39,854 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=087bf54664c6a240c659c71fb3782134, ASSIGN 2023-03-31 18:53:39,856 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=087bf54664c6a240c659c71fb3782134, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34951,1680288816339; forceNewPlan=false, retain=false 2023-03-31 18:53:40,007 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=087bf54664c6a240c659c71fb3782134, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:40,008 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288820007"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288820007"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288820007"}]},"ts":"1680288820007"} 2023-03-31 18:53:40,011 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:53:40,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-31 18:53:40,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:40,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 087bf54664c6a240c659c71fb3782134, NAME => 'TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:53:40,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:40,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:53:40,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:40,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:40,173 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:40,175 DEBUG [StoreOpener-087bf54664c6a240c659c71fb3782134-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A 2023-03-31 18:53:40,175 DEBUG [StoreOpener-087bf54664c6a240c659c71fb3782134-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A 2023-03-31 18:53:40,176 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:53:40,177 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 087bf54664c6a240c659c71fb3782134 columnFamilyName A 2023-03-31 18:53:40,178 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] regionserver.HStore(310): Store=087bf54664c6a240c659c71fb3782134/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:40,178 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:40,180 DEBUG [StoreOpener-087bf54664c6a240c659c71fb3782134-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B 2023-03-31 18:53:40,180 DEBUG [StoreOpener-087bf54664c6a240c659c71fb3782134-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B 2023-03-31 18:53:40,180 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:53:40,180 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 087bf54664c6a240c659c71fb3782134 columnFamilyName B 2023-03-31 18:53:40,181 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] regionserver.HStore(310): Store=087bf54664c6a240c659c71fb3782134/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:40,181 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:40,183 DEBUG [StoreOpener-087bf54664c6a240c659c71fb3782134-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C 2023-03-31 18:53:40,183 DEBUG [StoreOpener-087bf54664c6a240c659c71fb3782134-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C 2023-03-31 18:53:40,183 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:53:40,183 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 087bf54664c6a240c659c71fb3782134 columnFamilyName C 2023-03-31 18:53:40,184 INFO [StoreOpener-087bf54664c6a240c659c71fb3782134-1] regionserver.HStore(310): Store=087bf54664c6a240c659c71fb3782134/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:53:40,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:40,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:40,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:40,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:53:40,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:40,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 18:53:40,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 087bf54664c6a240c659c71fb3782134; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60441669, jitterRate=-0.09934894740581512}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:53:40,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:40,197 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., pid=11, masterSystemTime=1680288820164 2023-03-31 18:53:40,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:40,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:40,201 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=087bf54664c6a240c659c71fb3782134, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:40,201 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288820201"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680288820201"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680288820201"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680288820201"}]},"ts":"1680288820201"} 2023-03-31 18:53:40,207 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-31 18:53:40,207 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 in 193 msec 2023-03-31 18:53:40,210 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-31 18:53:40,211 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=087bf54664c6a240c659c71fb3782134, ASSIGN in 355 msec 2023-03-31 18:53:40,212 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 18:53:40,212 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288820212"}]},"ts":"1680288820212"} 2023-03-31 18:53:40,214 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 18:53:40,217 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 18:53:40,219 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 825 msec 2023-03-31 18:53:40,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-31 18:53:40,526 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-31 18:53:40,531 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x35594b1b to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:40,536 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55e5fd59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:40,539 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:40,543 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59984, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:40,547 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 18:53:40,551 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60662, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 18:53:40,559 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:40,564 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:40,564 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:40,574 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:40,575 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:40,579 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:40,579 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x63859c62 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:40,584 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c57e1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:40,584 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x42927198 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:40,588 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60b8c4c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:40,589 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:40,595 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:40,595 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:40,628 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:40,628 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x4b02eeb3 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:40,632 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63871a9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:40,633 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x4808c6d5 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:53:40,636 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1423ce11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:53:40,648 DEBUG [hconnection-0x4dbc39ec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:40,648 DEBUG [hconnection-0x38c73e86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:40,648 DEBUG [hconnection-0x57af7f7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:40,649 DEBUG [hconnection-0x106b4c0f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:40,649 DEBUG [hconnection-0x71b63d66-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:40,649 DEBUG [hconnection-0x44866d33-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:40,652 DEBUG [hconnection-0x30bca569-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:40,653 DEBUG [hconnection-0x2d61b265-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:40,653 DEBUG [hconnection-0x5ed56db8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:53:40,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:40,659 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59992, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:40,664 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59988, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:40,664 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:40,665 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60008, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:40,667 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:40,668 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:40,668 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:40,669 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:40,683 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60050, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:53:40,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:40,689 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:40,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:40,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:40,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:40,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:40,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:40,694 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:40,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:40,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:40,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:40,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:40,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:40,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:40,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:40,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:40,698 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:40,698 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:40,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:40,707 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:40,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:40,710 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-31 18:53:40,734 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:40,738 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:40,738 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:40,738 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:40,739 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:40,739 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:40,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:40,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288820837, max timestamp=1680288820856Num uniques 859; 2023-03-31 18:53:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:53:40,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288820837, max timestamp=1680288820886Num uniques 859; 2023-03-31 18:53:40,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:53:40,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288820837, max timestamp=1680288820856Num uniques 859; 2023-03-31 18:53:40,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 18:53:40,975 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/99d14cfdd76a40008da678a335636065 2023-03-31 18:53:41,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288820970, max timestamp=1680288820999Num uniques 1009; 2023-03-31 18:53:41,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-31 18:53:41,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288820970, max timestamp=1680288820999Num uniques 1018; 2023-03-31 18:53:41,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 18:53:41,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288820970, max timestamp=1680288820999Num uniques 1018; 2023-03-31 18:53:41,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 18:53:41,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1680288821111, max timestamp=1680288821126Num uniques 1027; 2023-03-31 18:53:41,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=10382, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-31 18:53:41,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680288821111, max timestamp=1680288821126Num uniques 1027; 2023-03-31 18:53:41,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-31 18:53:41,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288821111, max timestamp=1680288821126Num uniques 1027; 2023-03-31 18:53:41,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-03-31 18:53:41,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680288821160, max timestamp=1680288821175Num uniques 1086; 2023-03-31 18:53:41,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 18:53:41,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680288821160, max timestamp=1680288821174Num uniques 1009; 2023-03-31 18:53:41,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-03-31 18:53:41,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680288821160, max timestamp=1680288821174Num uniques 1086; 2023-03-31 18:53:41,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 18:53:41,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680288821213, max timestamp=1680288821224Num uniques 1045; 2023-03-31 18:53:41,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=8920, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-31 18:53:41,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288821213, max timestamp=1680288821224Num uniques 1045; 2023-03-31 18:53:41,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=6870, suffix heap size=136648,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=150 2023-03-31 18:53:41,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288821213, max timestamp=1680288821224Num uniques 1045; 2023-03-31 18:53:41,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=6870, suffix heap size=136648,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=150 2023-03-31 18:53:41,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680288821248, max timestamp=1680288821279Num uniques 1054; 2023-03-31 18:53:41,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-31 18:53:41,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680288821248, max timestamp=1680288821279Num uniques 1009; 2023-03-31 18:53:41,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=7050, suffix heap size=132904,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=154 2023-03-31 18:53:41,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288821248, max timestamp=1680288821279Num uniques 1009; 2023-03-31 18:53:41,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-31 18:53:41,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680288821338, max timestamp=1680288821352Num uniques 1013; 2023-03-31 18:53:41,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680288821338, max timestamp=1680288821352Num uniques 1013; 2023-03-31 18:53:41,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=14328, suffix heap size=133320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=313 2023-03-31 18:53:41,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288821346, max timestamp=1680288821352Num uniques 1009; 2023-03-31 18:53:41,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46208, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-31 18:53:41,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=14328, suffix heap size=133320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=313 2023-03-31 18:53:41,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1680288821366, max timestamp=1680288821377Num uniques 1172; 2023-03-31 18:53:41,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=14742, suffix heap size=149856,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=322 2023-03-31 18:53:41,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1680288821366, max timestamp=1680288821377Num uniques 1172; 2023-03-31 18:53:41,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1680288821366, max timestamp=1680288821378Num uniques 1009; 2023-03-31 18:53:41,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=14742, suffix heap size=132904,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=322 2023-03-31 18:53:41,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=14742, suffix heap size=149856,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=322 2023-03-31 18:53:41,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680288821395, max timestamp=1680288821406Num uniques 1181; 2023-03-31 18:53:41,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=12866, suffix heap size=150792,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=281 2023-03-31 18:53:41,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680288821395, max timestamp=1680288821406Num uniques 1181; 2023-03-31 18:53:41,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680288821395, max timestamp=1680288821406Num uniques 1181; 2023-03-31 18:53:41,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=10576, suffix heap size=150792,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=231 2023-03-31 18:53:41,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=10576, suffix heap size=150792,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=231 2023-03-31 18:53:41,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680288821432, max timestamp=1680288821443Num uniques 1090; 2023-03-31 18:53:41,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-31 18:53:41,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680288821432, max timestamp=1680288821443Num uniques 1090; 2023-03-31 18:53:41,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-31 18:53:41,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680288821432, max timestamp=1680288821443Num uniques 1140; 2023-03-31 18:53:41,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-31 18:53:41,474 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/6aef52c4376d46d2a48b7e2334e1611b 2023-03-31 18:53:41,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680288821464, max timestamp=1680288821477Num uniques 1049; 2023-03-31 18:53:41,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-31 18:53:41,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680288821464, max timestamp=1680288821480Num uniques 1049; 2023-03-31 18:53:41,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-31 18:53:41,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680288821464, max timestamp=1680288821477Num uniques 1049; 2023-03-31 18:53:41,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-31 18:53:41,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680288821512, max timestamp=1680288821521Num uniques 1058; 2023-03-31 18:53:41,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-31 18:53:41,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680288821512, max timestamp=1680288821521Num uniques 1058; 2023-03-31 18:53:41,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-31 18:53:41,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680288821512, max timestamp=1680288821522Num uniques 1058; 2023-03-31 18:53:41,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-31 18:53:41,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680288821545, max timestamp=1680288821557Num uniques 1017; 2023-03-31 18:53:41,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680288821545, max timestamp=1680288821557Num uniques 1017; 2023-03-31 18:53:41,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-31 18:53:41,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-31 18:53:41,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288821545, max timestamp=1680288821557Num uniques 1017; 2023-03-31 18:53:41,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=6870, suffix heap size=133736,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=150 2023-03-31 18:53:41,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680288821561, max timestamp=1680288821585Num uniques 1076; 2023-03-31 18:53:41,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-31 18:53:41,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680288821561, max timestamp=1680288821585Num uniques 1076; 2023-03-31 18:53:41,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-31 18:53:41,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680288821561, max timestamp=1680288821587Num uniques 1009; 2023-03-31 18:53:41,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8056, suffix heap size=132904,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=176 2023-03-31 18:53:41,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680288821596, max timestamp=1680288821610Num uniques 1035; 2023-03-31 18:53:41,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680288821596, max timestamp=1680288821610Num uniques 1035; 2023-03-31 18:53:41,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-31 18:53:41,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-31 18:53:41,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680288821596, max timestamp=1680288821610Num uniques 1035; 2023-03-31 18:53:41,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-31 18:53:41,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680288821631, max timestamp=1680288821636Num uniques 1094; 2023-03-31 18:53:41,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680288821631, max timestamp=1680288821636Num uniques 1094; 2023-03-31 18:53:41,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-31 18:53:41,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-31 18:53:41,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680288821631, max timestamp=1680288821636Num uniques 1044; 2023-03-31 18:53:41,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-31 18:53:41,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680288821685, max timestamp=1680288821695Num uniques 1053; 2023-03-31 18:53:41,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-03-31 18:53:41,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680288821685, max timestamp=1680288821696Num uniques 1053; 2023-03-31 18:53:41,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680288821685, max timestamp=1680288821696Num uniques 1053; 2023-03-31 18:53:41,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-03-31 18:53:41,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-03-31 18:53:41,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680288821714, max timestamp=1680288821724Num uniques 1062; 2023-03-31 18:53:41,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680288821714, max timestamp=1680288821724Num uniques 1062; 2023-03-31 18:53:41,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1680288821714, max timestamp=1680288821724Num uniques 1062; 2023-03-31 18:53:41,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-03-31 18:53:41,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-31 18:53:41,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-31 18:53:41,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1680288821747, max timestamp=1680288821754Num uniques 1071; 2023-03-31 18:53:41,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-03-31 18:53:41,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1680288821747, max timestamp=1680288821754Num uniques 1021; 2023-03-31 18:53:41,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-03-31 18:53:41,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1680288821747, max timestamp=1680288821754Num uniques 1021; 2023-03-31 18:53:41,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-03-31 18:53:41,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680288821786, max timestamp=1680288821830Num uniques 1080; 2023-03-31 18:53:41,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680288821786, max timestamp=1680288821830Num uniques 1080; 2023-03-31 18:53:41,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-03-31 18:53:41,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680288821786, max timestamp=1680288821830Num uniques 1080; 2023-03-31 18:53:41,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-03-31 18:53:41,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-03-31 18:53:41,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1680288821851, max timestamp=1680288821855Num uniques 1089; 2023-03-31 18:53:41,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-03-31 18:53:41,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1680288821851, max timestamp=1680288821855Num uniques 1089; 2023-03-31 18:53:41,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-03-31 18:53:41,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1680288821851, max timestamp=1680288821855Num uniques 1089; 2023-03-31 18:53:41,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-03-31 18:53:41,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680288821881, max timestamp=1680288821891Num uniques 1048; 2023-03-31 18:53:41,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288821887, max timestamp=1680288821892Num uniques 1048; 2023-03-31 18:53:41,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47988, new segment data size=6870, suffix heap size=136960,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=150 2023-03-31 18:53:41,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-03-31 18:53:41,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288821887, max timestamp=1680288821892Num uniques 1048; 2023-03-31 18:53:41,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47988, new segment data size=6870, suffix heap size=136960,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=150 2023-03-31 18:53:41,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9472, totalHeapSize=21768, min timestamp=1680288821917, max timestamp=1680288821927Num uniques 1107; 2023-03-31 18:53:41,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9472, totalHeapSize=21768, min timestamp=1680288821917, max timestamp=1680288821927Num uniques 1009; 2023-03-31 18:53:41,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=9472, suffix heap size=132904,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=207 2023-03-31 18:53:41,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50692, new segment data size=9472, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-03-31 18:53:41,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1680288821917, max timestamp=1680288821927Num uniques 1009; 2023-03-31 18:53:41,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46217, new segment data size=9475, suffix heap size=132904,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=207 2023-03-31 18:53:41,947 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/e8169e04ac1e40cab631c3d41f2f083d 2023-03-31 18:53:41,960 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/99d14cfdd76a40008da678a335636065 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/99d14cfdd76a40008da678a335636065 2023-03-31 18:53:41,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1680288821956, max timestamp=1680288821967Num uniques 1066; 2023-03-31 18:53:41,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=7596, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-03-31 18:53:41,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1680288821956, max timestamp=1680288821967Num uniques 1066; 2023-03-31 18:53:41,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=7596, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-03-31 18:53:41,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1680288821956, max timestamp=1680288821967Num uniques 1066; 2023-03-31 18:53:41,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=7596, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-03-31 18:53:41,981 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/99d14cfdd76a40008da678a335636065, entries=100, sequenceid=8, filesize=9.3 K 2023-03-31 18:53:41,990 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/6aef52c4376d46d2a48b7e2334e1611b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/6aef52c4376d46d2a48b7e2334e1611b 2023-03-31 18:53:42,011 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/6aef52c4376d46d2a48b7e2334e1611b, entries=100, sequenceid=8, filesize=9.3 K 2023-03-31 18:53:42,015 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/e8169e04ac1e40cab631c3d41f2f083d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/e8169e04ac1e40cab631c3d41f2f083d 2023-03-31 18:53:42,037 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/e8169e04ac1e40cab631c3d41f2f083d, entries=100, sequenceid=8, filesize=9.3 K 2023-03-31 18:53:42,042 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=100.63 KB/103050 for 087bf54664c6a240c659c71fb3782134 in 1332ms, sequenceid=8, compaction requested=false 2023-03-31 18:53:42,045 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-31 18:53:42,048 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:42,048 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:42,048 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:42,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:42,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:42,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:42,052 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,052 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:42,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2332f169[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:42,057 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:42,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:42,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=114.05 KB heapSize=324.56 KB 2023-03-31 18:53:42,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,067 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:42,067 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,067 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:42,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:42,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:42,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:42,067 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:42,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:42,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:42,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:42,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:42,074 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,074 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:42,075 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,075 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:42,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:42,081 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:42,081 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:42,081 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:42,081 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:42,081 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:42,094 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,095 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:42,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,097 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:42,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:42,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,118 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,119 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:42,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:42,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:42,120 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:42,120 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:42,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:42,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288882087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,122 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,122 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:42,122 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,122 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:42,122 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,122 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288882088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:42,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:42,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288882115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288882133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288882142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,222 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:42,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:42,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:42,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288882228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:42,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:42,248 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:42,248 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:42,250 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:42,250 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:42,251 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,251 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288882244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,253 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:42,253 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:42,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288882244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,253 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,254 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:42,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:42,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:42,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:42,264 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:42,264 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:42,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:42,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:42,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:42,272 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:42,272 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:42,272 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:42,272 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:42,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:42,274 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:42,274 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,274 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:42,274 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:42,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,278 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,280 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,280 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:42,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288882275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288882276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:42,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@246023e0[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:42,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:42,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,284 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:42,284 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:42,284 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:42,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:42,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:42,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:42,295 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:42,295 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:42,295 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:42,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:42,296 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:42,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:42,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,304 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,306 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,326 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:42,326 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:42,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:42,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:42,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:42,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:42,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:42,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:42,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:42,333 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:42,333 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:42,433 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:42,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:42,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:42,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:42,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:42,451 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:42,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:42,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:42,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:42,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288882450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,457 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:42,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:42,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:42,458 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,458 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:42,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288882457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288882457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:42,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:42,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:42,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:42,492 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:42,492 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:42,492 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:42,492 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:42,493 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:42,493 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:42,493 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:42,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:42,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:42,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:42,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:42,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:42,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e6d2d98[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:42,501 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:42,501 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,503 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,503 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:42,504 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,504 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,504 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:42,504 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:42,504 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:42,504 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:42,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288882501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288882504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,534 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:42,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:42,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:42,548 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:42,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:42,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:42,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:42,548 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:42,551 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,552 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,552 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:42,553 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,568 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,568 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:42,568 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:42,568 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:42,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:42,568 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,568 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,568 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,569 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:42,569 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:42,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:42,568 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:42,568 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:42,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:42,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:42,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/dca0460573744b9c97a83f8f2818d6b5 2023-03-31 18:53:42,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/b0d905e4bb294adc900219a0ec825dbd 2023-03-31 18:53:42,669 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:42,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:42,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:42,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:42,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:42,700 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:42,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:42,702 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:42,702 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:42,703 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,703 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:42,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:42,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:42,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:42,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:42,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:42,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:42,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:42,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:42,720 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:42,720 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:42,720 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:42,720 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:42,720 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:42,720 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:42,721 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:42,721 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:42,721 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,721 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:42,721 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:42,730 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,730 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:42,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f51a5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:42,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/52468637d50d456faac82725e2afaf86 2023-03-31 18:53:42,750 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:42,750 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,753 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,753 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:42,753 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:42,753 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:42,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,754 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:42,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:42,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288882767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288882767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:42,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:42,773 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:42,773 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:42,773 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:42,773 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:42,773 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:42,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288882768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/dca0460573744b9c97a83f8f2818d6b5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/dca0460573744b9c97a83f8f2818d6b5 2023-03-31 18:53:42,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:42,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:42,782 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:42,785 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:42,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/dca0460573744b9c97a83f8f2818d6b5, entries=450, sequenceid=437, filesize=26.2 K 2023-03-31 18:53:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/b0d905e4bb294adc900219a0ec825dbd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b0d905e4bb294adc900219a0ec825dbd 2023-03-31 18:53:42,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:42,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:42,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:42,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:42,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:42,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:42,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:42,797 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:42,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,797 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:42,799 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:42,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:42,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b0d905e4bb294adc900219a0ec825dbd, entries=150, sequenceid=437, filesize=11.9 K 2023-03-31 18:53:42,823 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/52468637d50d456faac82725e2afaf86 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/52468637d50d456faac82725e2afaf86 2023-03-31 18:53:42,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288882814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:42,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288882814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/52468637d50d456faac82725e2afaf86, entries=150, sequenceid=437, filesize=11.9 K 2023-03-31 18:53:42,848 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~324.52 KB/332304, currentSize=73.80 KB/75570 for 087bf54664c6a240c659c71fb3782134 in 782ms, sequenceid=437, compaction requested=false 2023-03-31 18:53:42,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:42,899 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:42,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:42,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:42,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:42,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:42,912 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:42,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:42,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:42,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:42,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:42,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:42,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:42,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:42,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:42,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:42,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:42,925 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:42,925 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:42,925 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:42,925 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:42,925 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-31 18:53:42,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:42,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:42,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:42,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:42,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:42,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:42,995 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/217d5f494fb041559ccde2d1895f273d 2023-03-31 18:53:43,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:43,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:43,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288883300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288883303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288883305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288883330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288883340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288883406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288883414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288883415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,479 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/d975dc679ba041f99c37d7daa8b41364 2023-03-31 18:53:43,534 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/409c65b545e946079f68ad1cac833a55 2023-03-31 18:53:43,546 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 18:53:43,550 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/217d5f494fb041559ccde2d1895f273d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/217d5f494fb041559ccde2d1895f273d 2023-03-31 18:53:43,581 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/217d5f494fb041559ccde2d1895f273d, entries=150, sequenceid=451, filesize=11.9 K 2023-03-31 18:53:43,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/d975dc679ba041f99c37d7daa8b41364 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/d975dc679ba041f99c37d7daa8b41364 2023-03-31 18:53:43,599 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/d975dc679ba041f99c37d7daa8b41364, entries=150, sequenceid=451, filesize=11.9 K 2023-03-31 18:53:43,601 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/409c65b545e946079f68ad1cac833a55 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/409c65b545e946079f68ad1cac833a55 2023-03-31 18:53:43,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288883612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,623 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/409c65b545e946079f68ad1cac833a55, entries=150, sequenceid=451, filesize=11.9 K 2023-03-31 18:53:43,629 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 087bf54664c6a240c659c71fb3782134 in 704ms, sequenceid=451, compaction requested=true 2023-03-31 18:53:43,629 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:43,630 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:43,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:43,630 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-31 18:53:43,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:43,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:43,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:43,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:43,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:43,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:43,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:43,635 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:43,636 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,636 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:43,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:43,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@319251c6[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:43,640 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:43,640 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,642 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,645 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:43,645 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:43,645 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:43,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:43,648 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:43,652 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:43,652 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:43,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:43,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:43,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:43,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:43,671 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:43,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:43,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:43,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:43,671 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:43,677 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,677 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,677 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:43,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:43,678 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:43,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:43,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,680 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,690 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:43,690 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:43,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:43,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:43,691 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,691 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:43,691 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:43,691 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,691 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,691 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,691 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,691 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:43,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:43,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:43,691 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:43,692 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:43,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:43,697 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-31 18:53:43,699 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-31 18:53:43,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/6b4f4bf25ea74dd09f4eb6baa8f6ea6f 2023-03-31 18:53:43,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288883712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288883714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,792 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:43,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:43,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:43,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:43,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:43,799 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:43,799 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:43,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:43,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:43,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:43,804 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,805 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:43,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:43,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:43,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:43,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:43,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:43,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:43,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:43,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:43,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:43,808 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:43,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:43,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:43,809 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:43,809 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:43,809 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:43,809 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:43,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:43,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:43,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:43,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,811 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:43,812 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,812 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:43,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:43,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14481b37[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:43,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:43,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:43,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:43,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:43,819 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,820 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:43,821 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:43,821 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:43,821 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:43,822 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288883824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288883824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:43,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:43,828 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:43,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:43,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:43,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:43,829 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:43,832 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,832 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,832 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:43,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:43,833 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,835 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,839 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:43,839 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:43,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:43,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:43,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:43,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:43,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:43,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:43,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:43,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:43,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,841 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,842 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:43,842 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:43,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:43,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288883919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,942 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:43,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:43,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:43,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:43,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:43,949 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:43,949 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:43,950 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:43,950 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:43,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:43,953 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,953 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:43,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:43,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:43,953 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,953 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:43,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:43,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:43,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:43,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:43,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:43,956 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:43,956 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:43,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:43,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:43,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:43,958 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:43,958 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:43,958 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:43,958 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:43,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:43,961 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,961 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,961 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:43,966 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:43,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:43,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732ef5cb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:43,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:43,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:43,973 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,973 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:43,973 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:43,973 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:43,973 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:43,975 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:43,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:43,976 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:43,976 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:43,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:43,982 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:43,982 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:43,982 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:43,982 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:43,982 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:43,987 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,987 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:43,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,988 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:43,990 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:43,995 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,005 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,005 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,005 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,005 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,005 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:44,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:44,005 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,008 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,008 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:44,008 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:44,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:44,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288884029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:44,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288884030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,109 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:44,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:44,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:44,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/b8e649b19a124969918d9c9174005cb4 2023-03-31 18:53:44,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:44,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:44,145 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:44,145 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:44,149 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:44,149 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:44,149 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,151 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,152 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,152 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:44,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:44,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:44,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:44,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:44,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:44,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:44,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:44,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:44,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:44,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:44,168 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:44,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc39072[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:44,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:44,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:44,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:44,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:44,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:44,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:44,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:44,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:44,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:44,195 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:44,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:44,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:44,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:44,195 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:44,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,198 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,199 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,208 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:44,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:44,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,208 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:44,208 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:44,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:44,309 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:44,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:44,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:44,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:44,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:44,315 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:44,315 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:44,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:44,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:44,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,319 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,319 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:44,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:44,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:44,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:44,331 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:44,332 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:44,332 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:44,332 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,332 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:44,332 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:44,332 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,332 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:44,332 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:44,332 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,332 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:44,332 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:44,335 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,335 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,335 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,335 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:44,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288884333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:44,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288884334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:44,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:44,338 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:44,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288884338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ffcfc4d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:44,338 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:44,342 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:44,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:44,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:44,346 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,347 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:44,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288884349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:44,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:44,353 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:44,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:44,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:44,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:44,353 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:44,355 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,355 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,355 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,356 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,358 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,361 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,361 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:44,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:44,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,362 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,363 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,363 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:44,363 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:44,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:44,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:44,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288884426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,463 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:44,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:44,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:44,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:44,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:44,468 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:44,468 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:44,469 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:44,470 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:44,470 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,470 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,472 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,472 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,472 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,472 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:44,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:44,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:44,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:44,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:44,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:44,477 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:44,477 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:44,477 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:44,477 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,478 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:44,478 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:44,478 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,478 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:44,478 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:44,478 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,478 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:44,478 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:44,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,481 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,482 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:44,483 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:44,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b39db26[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:44,483 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,485 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:44,485 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,485 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:44,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:44,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:44,487 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:44,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:44,492 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:44,492 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:44,492 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:44,492 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:44,492 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:44,494 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,494 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,494 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,495 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,497 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,502 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:44,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:44,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,502 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:44,502 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:44,504 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,504 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,504 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,504 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,504 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,504 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,504 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/88a6c6d7283742b0a7164cb42c8099c2 2023-03-31 18:53:44,602 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:44,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:44,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:44,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:44,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:44,609 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:44,609 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:44,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:44,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:44,611 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,611 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,613 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,613 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,614 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,614 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/6b4f4bf25ea74dd09f4eb6baa8f6ea6f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/6b4f4bf25ea74dd09f4eb6baa8f6ea6f 2023-03-31 18:53:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:44,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:44,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:44,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:44,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:44,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/6b4f4bf25ea74dd09f4eb6baa8f6ea6f, entries=250, sequenceid=471, filesize=16.7 K 2023-03-31 18:53:44,629 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:44,629 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:44,629 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:44,629 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,630 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:44,630 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:44,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:44,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:44,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/b8e649b19a124969918d9c9174005cb4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b8e649b19a124969918d9c9174005cb4 2023-03-31 18:53:44,633 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,633 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,634 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,634 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:44,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fbde927[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:44,637 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:44,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,640 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:44,640 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:44,640 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:44,640 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,640 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:44,641 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b8e649b19a124969918d9c9174005cb4, entries=150, sequenceid=471, filesize=11.9 K 2023-03-31 18:53:44,644 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/88a6c6d7283742b0a7164cb42c8099c2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/88a6c6d7283742b0a7164cb42c8099c2 2023-03-31 18:53:44,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:44,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:44,657 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:44,657 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:44,657 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:44,657 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:44,657 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:44,659 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,659 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,661 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,663 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/88a6c6d7283742b0a7164cb42c8099c2, entries=150, sequenceid=471, filesize=11.9 K 2023-03-31 18:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,666 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 087bf54664c6a240c659c71fb3782134 in 1036ms, sequenceid=471, compaction requested=true 2023-03-31 18:53:44,666 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:44,668 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,668 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,668 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,668 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:44,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,668 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,670 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:44,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,670 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:44,670 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,670 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:44,670 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:44,670 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:53:44,670 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:53:44,683 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:53:44,684 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 65624 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:53:44,684 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/A is initiating minor compaction (all files) 2023-03-31 18:53:44,684 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/B is initiating minor compaction (all files) 2023-03-31 18:53:44,685 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/A in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,685 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/B in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,685 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/6aef52c4376d46d2a48b7e2334e1611b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b0d905e4bb294adc900219a0ec825dbd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/d975dc679ba041f99c37d7daa8b41364, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b8e649b19a124969918d9c9174005cb4] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=45.0 K 2023-03-31 18:53:44,685 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/99d14cfdd76a40008da678a335636065, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/dca0460573744b9c97a83f8f2818d6b5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/217d5f494fb041559ccde2d1895f273d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/6b4f4bf25ea74dd09f4eb6baa8f6ea6f] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=64.1 K 2023-03-31 18:53:44,686 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 99d14cfdd76a40008da678a335636065, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680288820691 2023-03-31 18:53:44,686 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 6aef52c4376d46d2a48b7e2334e1611b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680288820691 2023-03-31 18:53:44,690 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting dca0460573744b9c97a83f8f2818d6b5, keycount=450, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680288821974 2023-03-31 18:53:44,691 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b0d905e4bb294adc900219a0ec825dbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680288821974 2023-03-31 18:53:44,691 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 217d5f494fb041559ccde2d1895f273d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1680288822085 2023-03-31 18:53:44,692 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d975dc679ba041f99c37d7daa8b41364, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1680288822085 2023-03-31 18:53:44,692 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 6b4f4bf25ea74dd09f4eb6baa8f6ea6f, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680288823277 2023-03-31 18:53:44,693 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b8e649b19a124969918d9c9174005cb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680288823303 2023-03-31 18:53:44,735 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#B#compaction#12 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:44,750 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#A#compaction#13 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:44,770 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:44,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:44,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:44,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:44,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:44,816 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:44,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:44,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:44,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:44,833 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,838 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,846 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,846 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,846 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,846 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:44,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:44,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:44,849 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:44,849 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:44,851 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:44,851 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:44,852 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,854 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-31 18:53:44,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:44,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:44,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:44,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:44,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:44,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:44,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:44,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:44,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:44,856 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:44,861 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:44,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6db6a2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:44,865 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:44,865 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,867 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:44,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:44,870 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:44,870 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,870 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:44,870 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:44,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:44,875 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:44,875 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:44,875 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:44,875 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:44,875 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:44,878 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,878 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:44,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,878 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:44,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:44,889 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,894 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,894 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,894 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:44,894 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:44,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:44,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:44,895 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,896 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,896 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:44,896 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:44,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:44,896 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:44,896 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:44,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:44,897 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:44,897 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:44,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/c33bc7398d7e462289b598635265c7f1 2023-03-31 18:53:44,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:44,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60050 deadline: 1680288884921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:44,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59988 deadline: 1680288884924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:44,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/73ec365395de46d589c250c183bce43d 2023-03-31 18:53:44,997 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:45,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:45,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:45,007 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/66d85dbb216c431b96806e4287d38c5e 2023-03-31 18:53:45,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/c33bc7398d7e462289b598635265c7f1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/c33bc7398d7e462289b598635265c7f1 2023-03-31 18:53:45,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:45,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:45,020 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:45,021 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:45,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:45,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:45,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:45,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:45,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:45,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:45,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:45,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:45,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:45,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:45,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:45,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288885030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:45,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:45,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288885032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:45,036 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,040 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/c33bc7398d7e462289b598635265c7f1, entries=150, sequenceid=486, filesize=11.9 K 2023-03-31 18:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/73ec365395de46d589c250c183bce43d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/73ec365395de46d589c250c183bce43d 2023-03-31 18:53:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:45,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@273b3196[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:45,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:45,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:45,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:45,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:45,057 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,057 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:45,059 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,067 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/73ec365395de46d589c250c183bce43d, entries=150, sequenceid=486, filesize=11.9 K 2023-03-31 18:53:45,073 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/66d85dbb216c431b96806e4287d38c5e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/66d85dbb216c431b96806e4287d38c5e 2023-03-31 18:53:45,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:45,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:45,084 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:45,084 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:45,084 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:45,084 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:45,084 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:45,087 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,092 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,093 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/66d85dbb216c431b96806e4287d38c5e, entries=150, sequenceid=486, filesize=11.9 K 2023-03-31 18:53:45,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 087bf54664c6a240c659c71fb3782134 in 253ms, sequenceid=486, compaction requested=true 2023-03-31 18:53:45,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:45,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 18:53:45,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-31 18:53:45,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-31 18:53:45,107 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,107 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,107 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,107 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,113 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,113 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,113 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,113 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:45,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:45,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:45,114 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:45,114 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:45,114 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,114 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,214 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:45,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:45,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:45,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:45,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:45,220 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:45,220 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,221 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:45,221 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:45,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,225 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,226 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,227 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,227 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:45,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:45,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:45,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:45,235 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:45,235 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,236 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:45,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:45,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-31 18:53:45,240 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:45,240 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:45,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:45,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:45,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:45,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:45,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:45,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:45,244 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,244 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:45,244 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:45,244 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:45,249 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,249 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,250 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,250 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12ee3ebc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:45,256 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:45,256 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,260 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:45,260 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,260 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:45,264 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:45,264 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,264 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:45,264 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,266 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/a33f517bb759403798a5835293835d2a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a33f517bb759403798a5835293835d2a 2023-03-31 18:53:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:45,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:45,270 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:45,270 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:45,270 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:45,270 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:45,270 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:45,273 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,273 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,273 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,274 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,275 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-31 18:53:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,276 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-31 18:53:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288885275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288885278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,286 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,286 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,287 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,287 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,287 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:45,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:45,287 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,287 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,287 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:45,287 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/af6bd5c42ed4426b96c69327d6355be3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/af6bd5c42ed4426b96c69327d6355be3 2023-03-31 18:53:45,287 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,288 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,287 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:45,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:45,320 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 087bf54664c6a240c659c71fb3782134/A of 087bf54664c6a240c659c71fb3782134 into a33f517bb759403798a5835293835d2a(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:45,320 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 087bf54664c6a240c659c71fb3782134/B of 087bf54664c6a240c659c71fb3782134 into af6bd5c42ed4426b96c69327d6355be3(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:45,320 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:45,320 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/A, priority=12, startTime=1680288824669; duration=0sec 2023-03-31 18:53:45,320 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:45,320 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/B, priority=12, startTime=1680288824670; duration=0sec 2023-03-31 18:53:45,321 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-31 18:53:45,321 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-31 18:53:45,321 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:53:45,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/727adf9ff5934ea39f3797f3dbcaad33 2023-03-31 18:53:45,333 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58286 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:53:45,333 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/C is initiating minor compaction (all files) 2023-03-31 18:53:45,333 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-03-31 18:53:45,334 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/C in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,334 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 18:53:45,334 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 18:53:45,334 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. because compaction request was cancelled 2023-03-31 18:53:45,334 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/e8169e04ac1e40cab631c3d41f2f083d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/52468637d50d456faac82725e2afaf86, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/409c65b545e946079f68ad1cac833a55, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/88a6c6d7283742b0a7164cb42c8099c2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/66d85dbb216c431b96806e4287d38c5e] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=56.9 K 2023-03-31 18:53:45,334 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 18:53:45,335 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e8169e04ac1e40cab631c3d41f2f083d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680288820691 2023-03-31 18:53:45,348 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 18:53:45,349 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 18:53:45,349 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 52468637d50d456faac82725e2afaf86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680288821974 2023-03-31 18:53:45,349 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. because compaction request was cancelled 2023-03-31 18:53:45,349 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 18:53:45,350 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 409c65b545e946079f68ad1cac833a55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1680288822085 2023-03-31 18:53:45,350 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 18:53:45,350 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 18:53:45,351 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. because compaction request was cancelled 2023-03-31 18:53:45,351 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 88a6c6d7283742b0a7164cb42c8099c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680288823303 2023-03-31 18:53:45,352 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 66d85dbb216c431b96806e4287d38c5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680288823698 2023-03-31 18:53:45,384 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#C#compaction#19 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:45,388 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:45,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288885381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:45,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:45,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288885409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288885430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:45,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:45,444 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:45,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:45,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:45,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,456 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:45,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:45,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:45,459 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:45,459 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,462 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:45,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:45,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:45,464 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,464 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:45,464 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:45,464 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:45,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:45,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:45,465 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:45,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,467 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:45,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a1917[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:45,475 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:45,475 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,480 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:45,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:45,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:45,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:45,482 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:45,482 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,482 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,483 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:45,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:45,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:45,487 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:45,487 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:45,487 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:45,487 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:45,487 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:45,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,491 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,493 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:45,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,498 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:45,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:45,501 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:45,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,504 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:45,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288885591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,605 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:45,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:45,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:45,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:45,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:45,612 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:45,612 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,613 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:45,613 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:45,614 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,614 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288885612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,615 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,616 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:45,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:45,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:45,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:45,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:45,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:45,620 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,620 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:45,620 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:45,621 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:45,623 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,623 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:45,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d06752[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:45,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:45,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,627 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:45,629 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,629 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:45,631 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:45,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:45,635 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:45,635 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:45,635 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:45,635 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:45,635 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:45,637 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,637 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,637 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,644 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,647 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,648 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:45,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,649 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:45,649 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:45,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:45,749 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:45,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:45,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:45,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:45,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:45,754 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:45,754 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:45,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:45,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,757 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:45,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:45,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:45,758 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:45,758 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:45,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:45,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:45,759 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:45,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:45,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:45,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:45,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:45,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:45,762 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,762 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:45,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e99515f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:45,770 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:45,770 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,771 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:45,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:45,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:45,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:45,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,773 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:45,773 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:45,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:45,782 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:45,783 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:45,783 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:45,783 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:45,783 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:45,785 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,785 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,785 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,787 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:45,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:45,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:45,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,796 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:45,797 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:45,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/945b63273535445aa65ce78146c534f7 2023-03-31 18:53:45,882 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/348d0e867f9345d5942a2c32b9cd4ae9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/348d0e867f9345d5942a2c32b9cd4ae9 2023-03-31 18:53:45,893 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 087bf54664c6a240c659c71fb3782134/C of 087bf54664c6a240c659c71fb3782134 into 348d0e867f9345d5942a2c32b9cd4ae9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:45,893 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:45,893 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/C, priority=11, startTime=1680288825107; duration=0sec 2023-03-31 18:53:45,893 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:45,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288885895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,899 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:45,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:45,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:45,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:45,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:45,908 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:45,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:45,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:45,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,915 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:45,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:45,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:45,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:45,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:45,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:45,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:45,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:45,918 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,918 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:45,919 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:45,919 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:45,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:45,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288885919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,921 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,924 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:45,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de18bee[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:45,924 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:45,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:45,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:45,926 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,926 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:45,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:45,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:45,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:45,935 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:45,936 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:45,936 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:45,936 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:45,936 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:45,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:45,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:45,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,945 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,950 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:45,950 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:45,950 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,950 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,950 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:45,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:45,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:45,950 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:45,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:45,950 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:45,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:45,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:45,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:45,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:45,952 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:45,952 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:46,052 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:46,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:46,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:46,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:46,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:46,064 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:46,064 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:46,068 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:46,068 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:46,069 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,069 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,070 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,071 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,071 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:46,071 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:46,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:46,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:46,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:46,080 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:46,080 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:46,082 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:46,084 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:46,084 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:46,084 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:46,084 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:46,084 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:46,084 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:46,084 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:46,084 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:46,084 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,084 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:46,084 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:46,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:46,089 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,089 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f71fbe4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:46,092 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:46,092 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,095 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,095 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:46,095 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:46,095 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:46,100 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,106 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:46,107 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:46,107 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:46,107 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:46,107 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:46,107 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:46,107 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:46,111 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,111 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,111 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:46,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:46,116 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:46,119 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,123 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:46,123 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:46,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:46,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,124 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:46,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,129 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:46,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:46,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:46,129 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:46,230 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:46,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:46,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:46,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:46,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:46,243 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:46,243 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:46,244 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:46,244 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:46,245 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,245 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,247 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:46,247 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:46,247 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,247 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:46,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:46,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:46,256 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:46,256 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:46,256 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:46,256 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:46,258 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:46,259 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:46,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:46,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:46,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c18ffaf[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:46,261 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:46,261 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,263 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,264 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:46,264 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:46,264 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:46,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:46,267 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:46,267 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:46,267 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:46,270 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:46,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:46,274 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:46,274 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:46,274 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:46,274 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:46,274 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:46,279 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,279 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,279 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:46,281 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:46,286 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:46,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:46,286 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:46,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:46,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:46,287 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,287 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:46,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,290 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:46,290 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:46,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/41ecd0957d824bb79b2f7501342a899f 2023-03-31 18:53:46,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/727adf9ff5934ea39f3797f3dbcaad33 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/727adf9ff5934ea39f3797f3dbcaad33 2023-03-31 18:53:46,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/727adf9ff5934ea39f3797f3dbcaad33, entries=250, sequenceid=505, filesize=16.7 K 2023-03-31 18:53:46,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/945b63273535445aa65ce78146c534f7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/945b63273535445aa65ce78146c534f7 2023-03-31 18:53:46,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/945b63273535445aa65ce78146c534f7, entries=150, sequenceid=505, filesize=11.9 K 2023-03-31 18:53:46,343 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/41ecd0957d824bb79b2f7501342a899f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/41ecd0957d824bb79b2f7501342a899f 2023-03-31 18:53:46,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:46,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288886344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,357 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4209 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:53:46,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:46,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288886358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/41ecd0957d824bb79b2f7501342a899f, entries=150, sequenceid=505, filesize=11.9 K 2023-03-31 18:53:46,370 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4241 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:53:46,371 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 087bf54664c6a240c659c71fb3782134 in 1130ms, sequenceid=505, compaction requested=true 2023-03-31 18:53:46,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:46,371 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:46,371 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:46,372 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:46,372 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:46,372 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:46,375 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41576 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:46,375 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/A is initiating minor compaction (all files) 2023-03-31 18:53:46,375 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/A in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:46,375 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a33f517bb759403798a5835293835d2a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/c33bc7398d7e462289b598635265c7f1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/727adf9ff5934ea39f3797f3dbcaad33] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=40.6 K 2023-03-31 18:53:46,376 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a33f517bb759403798a5835293835d2a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680288823303 2023-03-31 18:53:46,377 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c33bc7398d7e462289b598635265c7f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680288823698 2023-03-31 18:53:46,378 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 727adf9ff5934ea39f3797f3dbcaad33, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680288824911 2023-03-31 18:53:46,379 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:46,385 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/B is initiating minor compaction (all files) 2023-03-31 18:53:46,385 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/B in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:46,385 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/af6bd5c42ed4426b96c69327d6355be3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/73ec365395de46d589c250c183bce43d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/945b63273535445aa65ce78146c534f7] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=35.8 K 2023-03-31 18:53:46,386 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting af6bd5c42ed4426b96c69327d6355be3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680288823303 2023-03-31 18:53:46,387 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 73ec365395de46d589c250c183bce43d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680288823698 2023-03-31 18:53:46,387 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 945b63273535445aa65ce78146c534f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680288824911 2023-03-31 18:53:46,390 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:46,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:46,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:46,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:46,399 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:46,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:46,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:46,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:46,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,402 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,403 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,403 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:46,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:46,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:46,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:46,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:46,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:46,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:46,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:46,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:46,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:46,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:46,414 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:46,414 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#B#compaction#22 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:46,414 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#A#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:46,414 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:46,415 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 18:53:46,415 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:46,415 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:46,415 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:46,415 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:46,415 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:46,415 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:46,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:46,428 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:46,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:46,478 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/695915fe34d14e7da4a77f36c8fe6e06 2023-03-31 18:53:46,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288886476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:46,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288886479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,504 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/58c16491d6a04565b79fe0f88473dfe3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/58c16491d6a04565b79fe0f88473dfe3 2023-03-31 18:53:46,515 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/A of 087bf54664c6a240c659c71fb3782134 into 58c16491d6a04565b79fe0f88473dfe3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:46,516 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:46,516 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/A, priority=13, startTime=1680288826371; duration=0sec 2023-03-31 18:53:46,516 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:46,516 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 18:53:46,523 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 18:53:46,523 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 18:53:46,523 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. because compaction request was cancelled 2023-03-31 18:53:46,543 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/1d3f12ba99324e539621a40982481351 2023-03-31 18:53:46,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:46,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288886580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:46,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288886584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:46,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288886785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:46,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288886789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:46,881 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/3bffb0c94c0d44788409ed6e75f85c0c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3bffb0c94c0d44788409ed6e75f85c0c 2023-03-31 18:53:46,889 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/B of 087bf54664c6a240c659c71fb3782134 into 3bffb0c94c0d44788409ed6e75f85c0c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:46,889 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:46,889 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/B, priority=13, startTime=1680288826371; duration=0sec 2023-03-31 18:53:46,889 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:46,971 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/4d1602af55a244afb4d0966ab455d036 2023-03-31 18:53:46,981 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/695915fe34d14e7da4a77f36c8fe6e06 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/695915fe34d14e7da4a77f36c8fe6e06 2023-03-31 18:53:46,991 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/695915fe34d14e7da4a77f36c8fe6e06, entries=150, sequenceid=521, filesize=11.9 K 2023-03-31 18:53:46,994 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/1d3f12ba99324e539621a40982481351 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/1d3f12ba99324e539621a40982481351 2023-03-31 18:53:47,005 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/1d3f12ba99324e539621a40982481351, entries=150, sequenceid=521, filesize=11.9 K 2023-03-31 18:53:47,008 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/4d1602af55a244afb4d0966ab455d036 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/4d1602af55a244afb4d0966ab455d036 2023-03-31 18:53:47,017 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/4d1602af55a244afb4d0966ab455d036, entries=150, sequenceid=521, filesize=11.9 K 2023-03-31 18:53:47,019 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 087bf54664c6a240c659c71fb3782134 in 604ms, sequenceid=521, compaction requested=true 2023-03-31 18:53:47,019 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:47,020 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:47,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:47,021 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,021 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:47,021 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:47,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@783f4cda[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:47,027 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:47,027 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:47,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:47,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:47,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:47,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:47,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:47,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:47,043 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:47,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:47,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:47,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:47,044 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:47,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:47,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:47,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:47,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:47,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:47,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,060 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:47,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:47,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:47,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:47,061 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:47,061 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:47,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:47,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:47,093 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:53:47,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:47,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:47,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:47,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:47,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:47,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:47,127 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/2e36e654f6af4d87a51b6aa238274ba4 2023-03-31 18:53:47,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288887126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288887126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:47,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:47,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:47,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:47,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:47,172 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:47,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:47,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:47,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:47,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:47,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:47,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:47,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:47,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:47,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:47,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:47,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:47,208 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,208 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:47,208 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:47,208 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:47,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:47,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:47,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:47,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:47,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:47,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@691006c8[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:47,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:47,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,215 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,215 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:47,215 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:47,215 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:47,215 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,215 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:47,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:47,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:47,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:47,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:47,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:47,225 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:47,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:47,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:47,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:47,225 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:47,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:47,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,230 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288887232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,234 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288887233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,246 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:47,246 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:47,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:47,246 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,246 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,246 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,246 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,246 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:47,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:47,246 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,246 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:47,247 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,247 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:47,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:47,247 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:47,347 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:47,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:47,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:47,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:47,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:47,351 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:47,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:47,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:47,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:47,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,354 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:47,354 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:47,354 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,354 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:47,356 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,356 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:47,356 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:47,356 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:47,358 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,358 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,359 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,359 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:47,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29fe6ed5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:47,361 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:47,361 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,364 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:47,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:47,366 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:47,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:47,367 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:47,367 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:47,367 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:47,367 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:47,367 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:47,367 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:47,367 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:47,368 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,368 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,370 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,371 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:47,375 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:47,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:47,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:47,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:47,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:47,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:47,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,376 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:47,376 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,376 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:47,376 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:47,376 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288887436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288887436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288887450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,454 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:53:47,476 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:47,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:47,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:47,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:47,481 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:47,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:47,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:47,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:47,482 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,482 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:47,484 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:47,485 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,485 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:47,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:47,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:47,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:47,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:47,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:47,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:47,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:47,487 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,487 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:47,487 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:47,488 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:47,490 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,490 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:47,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@229019c2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:47,493 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:47,493 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,494 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:47,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:47,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:47,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:47,496 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:47,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:47,499 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:47,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:47,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:47,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:47,499 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:47,502 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,502 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:47,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:47,502 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:47,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:47,503 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:47,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:47,510 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:47,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:47,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:47,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:47,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:47,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:47,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,511 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:47,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:47,511 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:47,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:47,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/23562af769be4e158c6b50cd797b3255 2023-03-31 18:53:47,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/f7815818dfe8490b89e8650d70ee8fae 2023-03-31 18:53:47,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/2e36e654f6af4d87a51b6aa238274ba4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2e36e654f6af4d87a51b6aa238274ba4 2023-03-31 18:53:47,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2e36e654f6af4d87a51b6aa238274ba4, entries=200, sequenceid=543, filesize=14.3 K 2023-03-31 18:53:47,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/23562af769be4e158c6b50cd797b3255 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/23562af769be4e158c6b50cd797b3255 2023-03-31 18:53:47,607 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/23562af769be4e158c6b50cd797b3255, entries=150, sequenceid=543, filesize=11.9 K 2023-03-31 18:53:47,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/f7815818dfe8490b89e8650d70ee8fae as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/f7815818dfe8490b89e8650d70ee8fae 2023-03-31 18:53:47,614 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:47,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:47,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:47,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:47,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:47,628 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:47,628 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:47,628 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:47,628 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:47,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,631 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/f7815818dfe8490b89e8650d70ee8fae, entries=150, sequenceid=543, filesize=11.9 K 2023-03-31 18:53:47,633 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,633 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:47,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:47,634 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,634 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:47,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:47,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 087bf54664c6a240c659c71fb3782134 in 543ms, sequenceid=543, compaction requested=true 2023-03-31 18:53:47,636 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:47,636 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:47,636 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:47,636 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:47,636 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:47,636 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:47,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:47,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:47,638 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:47,638 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/A is initiating minor compaction (all files) 2023-03-31 18:53:47,638 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/A in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,638 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:47,639 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/58c16491d6a04565b79fe0f88473dfe3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/695915fe34d14e7da4a77f36c8fe6e06, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2e36e654f6af4d87a51b6aa238274ba4] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=38.3 K 2023-03-31 18:53:47,639 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/B is initiating minor compaction (all files) 2023-03-31 18:53:47,639 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/B in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,639 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3bffb0c94c0d44788409ed6e75f85c0c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/1d3f12ba99324e539621a40982481351, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/23562af769be4e158c6b50cd797b3255] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=35.9 K 2023-03-31 18:53:47,640 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 58c16491d6a04565b79fe0f88473dfe3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680288824911 2023-03-31 18:53:47,647 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:47,640 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3bffb0c94c0d44788409ed6e75f85c0c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680288824911 2023-03-31 18:53:47,647 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,648 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:47,649 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:53:47,649 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 695915fe34d14e7da4a77f36c8fe6e06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680288825266 2023-03-31 18:53:47,650 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 2e36e654f6af4d87a51b6aa238274ba4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1680288826450 2023-03-31 18:53:47,651 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 1d3f12ba99324e539621a40982481351, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680288825266 2023-03-31 18:53:47,652 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 23562af769be4e158c6b50cd797b3255, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1680288826471 2023-03-31 18:53:47,657 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:47,657 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:47,657 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:47,657 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:47,657 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:47,657 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:47,682 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#A#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:47,694 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#B#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:47,720 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/8a122b3bc14e4649bfbd44cb50590cdc 2023-03-31 18:53:47,736 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/2255640946054dd1bc8d1b4237795e6f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2255640946054dd1bc8d1b4237795e6f 2023-03-31 18:53:47,742 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:47,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:47,747 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/B of 087bf54664c6a240c659c71fb3782134 into 2255640946054dd1bc8d1b4237795e6f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:47,747 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:47,747 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/B, priority=13, startTime=1680288827636; duration=0sec 2023-03-31 18:53:47,747 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:47,747 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:53:47,751 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48916 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:53:47,751 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/C is initiating minor compaction (all files) 2023-03-31 18:53:47,751 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/C in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:47,751 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/348d0e867f9345d5942a2c32b9cd4ae9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/41ecd0957d824bb79b2f7501342a899f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/4d1602af55a244afb4d0966ab455d036, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/f7815818dfe8490b89e8650d70ee8fae] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=47.8 K 2023-03-31 18:53:47,752 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 348d0e867f9345d5942a2c32b9cd4ae9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680288823698 2023-03-31 18:53:47,753 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 41ecd0957d824bb79b2f7501342a899f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680288824911 2023-03-31 18:53:47,753 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4d1602af55a244afb4d0966ab455d036, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680288825266 2023-03-31 18:53:47,754 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f7815818dfe8490b89e8650d70ee8fae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1680288826471 2023-03-31 18:53:47,769 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#C#compaction#33 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:47,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288887805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288887806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288887913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:47,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:47,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288887913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:48,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288888116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:48,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288888117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,127 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/3c78df96cefb47758fdf5e4379aa1d69 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/3c78df96cefb47758fdf5e4379aa1d69 2023-03-31 18:53:48,137 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/A of 087bf54664c6a240c659c71fb3782134 into 3c78df96cefb47758fdf5e4379aa1d69(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:48,137 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:48,137 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/A, priority=13, startTime=1680288827636; duration=0sec 2023-03-31 18:53:48,137 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:48,149 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/f84e4521b2244e8f9b6ffb68a6a5d53f 2023-03-31 18:53:48,188 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/8a94c494b0b4470bafaf6bb797e2b237 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8a94c494b0b4470bafaf6bb797e2b237 2023-03-31 18:53:48,197 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 087bf54664c6a240c659c71fb3782134/C of 087bf54664c6a240c659c71fb3782134 into 8a94c494b0b4470bafaf6bb797e2b237(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:48,198 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:48,198 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/C, priority=12, startTime=1680288827636; duration=0sec 2023-03-31 18:53:48,198 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:48,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:48,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288888419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:48,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288888419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,576 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/ec885308c4f5487b93003a43d84d56cd 2023-03-31 18:53:48,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/8a122b3bc14e4649bfbd44cb50590cdc as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/8a122b3bc14e4649bfbd44cb50590cdc 2023-03-31 18:53:48,592 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/8a122b3bc14e4649bfbd44cb50590cdc, entries=100, sequenceid=555, filesize=9.5 K 2023-03-31 18:53:48,593 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/f84e4521b2244e8f9b6ffb68a6a5d53f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f84e4521b2244e8f9b6ffb68a6a5d53f 2023-03-31 18:53:48,601 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f84e4521b2244e8f9b6ffb68a6a5d53f, entries=100, sequenceid=555, filesize=9.5 K 2023-03-31 18:53:48,607 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/ec885308c4f5487b93003a43d84d56cd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ec885308c4f5487b93003a43d84d56cd 2023-03-31 18:53:48,621 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ec885308c4f5487b93003a43d84d56cd, entries=100, sequenceid=555, filesize=9.5 K 2023-03-31 18:53:48,623 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 087bf54664c6a240c659c71fb3782134 in 974ms, sequenceid=555, compaction requested=false 2023-03-31 18:53:48,623 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:48,623 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:48,623 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:48,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:48,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:48,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:48,627 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:48,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:48,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@204bd8d6[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:48,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:48,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:48,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:48,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:48,633 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,633 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:48,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:48,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:48,639 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:48,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:48,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:48,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:48,639 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:48,642 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:48,642 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:48,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:48,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:48,643 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:48,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,646 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:48,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:48,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:48,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:48,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:48,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:48,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:48,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:48,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:48,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:48,651 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:48,651 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:48,651 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:48,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:48,751 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:48,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:48,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:48,757 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:48,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:48,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:48,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:48,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:48,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:48,759 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,759 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:48,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:48,759 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,759 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:48,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:48,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:48,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:48,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:48,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:48,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:48,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:48,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:48,761 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:48,761 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:48,761 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:53:48,762 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:48,762 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:48,762 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:48,762 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:48,762 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:48,762 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:48,772 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/87ca19054e6b4be1ac516a1637aac3f4 2023-03-31 18:53:48,791 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/f3d009ec29c54c9f906c9e2bf4c79346 2023-03-31 18:53:48,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:48,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:48,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:48,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288888935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:48,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:48,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288888937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:49,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288889039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:49,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288889042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,218 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/9967b5ae49fa4ebaad3882af54dc7225 2023-03-31 18:53:49,225 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/87ca19054e6b4be1ac516a1637aac3f4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/87ca19054e6b4be1ac516a1637aac3f4 2023-03-31 18:53:49,232 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/87ca19054e6b4be1ac516a1637aac3f4, entries=150, sequenceid=578, filesize=11.9 K 2023-03-31 18:53:49,235 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/f3d009ec29c54c9f906c9e2bf4c79346 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f3d009ec29c54c9f906c9e2bf4c79346 2023-03-31 18:53:49,244 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f3d009ec29c54c9f906c9e2bf4c79346, entries=150, sequenceid=578, filesize=11.9 K 2023-03-31 18:53:49,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:49,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288889248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,253 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/9967b5ae49fa4ebaad3882af54dc7225 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/9967b5ae49fa4ebaad3882af54dc7225 2023-03-31 18:53:49,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:49,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288889253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,261 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/9967b5ae49fa4ebaad3882af54dc7225, entries=150, sequenceid=578, filesize=11.9 K 2023-03-31 18:53:49,263 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 087bf54664c6a240c659c71fb3782134 in 502ms, sequenceid=578, compaction requested=true 2023-03-31 18:53:49,263 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:49,263 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:49,266 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4904ac51[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:49,268 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:49,268 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,269 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:49,270 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,270 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:49,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:49,275 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:49,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:49,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:49,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:49,275 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:49,278 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,278 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,278 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:49,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,280 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:49,281 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:49,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:49,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:49,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:49,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:49,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:49,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:49,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:49,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:49,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:49,285 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:49,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,285 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:49,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:49,385 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:49,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:49,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:49,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:49,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:49,389 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:49,389 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:49,389 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:49,390 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:49,390 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,390 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:49,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:49,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:49,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:49,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:49,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:49,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:49,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:49,393 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:49,393 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:49,394 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:49,394 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:49,394 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:53:49,394 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:49,394 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:49,395 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:49,395 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:49,395 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:49,395 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:49,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:49,556 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:49,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:49,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288889598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:49,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288889601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:49,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288889704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:49,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288889707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,806 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/bb532030c24d41f1b0300a46ffe8ddf6 2023-03-31 18:53:49,834 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/4a20eb53f94b44fe82768e40185e86c6 2023-03-31 18:53:49,853 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/b2645fcfed0a43b3a25ee047389a2736 2023-03-31 18:53:49,859 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/bb532030c24d41f1b0300a46ffe8ddf6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/bb532030c24d41f1b0300a46ffe8ddf6 2023-03-31 18:53:49,865 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/bb532030c24d41f1b0300a46ffe8ddf6, entries=150, sequenceid=590, filesize=11.9 K 2023-03-31 18:53:49,866 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/4a20eb53f94b44fe82768e40185e86c6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/4a20eb53f94b44fe82768e40185e86c6 2023-03-31 18:53:49,872 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/4a20eb53f94b44fe82768e40185e86c6, entries=150, sequenceid=590, filesize=11.9 K 2023-03-31 18:53:49,873 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/b2645fcfed0a43b3a25ee047389a2736 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/b2645fcfed0a43b3a25ee047389a2736 2023-03-31 18:53:49,886 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/b2645fcfed0a43b3a25ee047389a2736, entries=150, sequenceid=590, filesize=11.9 K 2023-03-31 18:53:49,887 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 087bf54664c6a240c659c71fb3782134 in 493ms, sequenceid=590, compaction requested=true 2023-03-31 18:53:49,888 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:49,888 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:49,891 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:49,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:49,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:49,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d6fb62[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:49,895 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:49,896 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,896 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:49,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:49,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:49,903 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:49,903 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:49,903 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:49,903 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:49,903 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:49,906 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,906 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:49,906 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:49,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288829592, max timestamp=1680288829910Num uniques 859; 2023-03-31 18:53:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288829592, max timestamp=1680288829910Num uniques 859; 2023-03-31 18:53:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:53:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:53:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288829592, max timestamp=1680288829910Num uniques 859; 2023-03-31 18:53:49,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:53:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:49,913 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=26.84 KB heapSize=66.23 KB 2023-03-31 18:53:49,913 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:49,913 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:49,913 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:49,913 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:49,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:49,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:49,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:49,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:49,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:49,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:49,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:49,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:49,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:49,915 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:49,915 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:49,915 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:49,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:49,915 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,915 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:49,915 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:49,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:49,920 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:49,920 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:49,948 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/cd3d4c3d58cc485e935489f002efad72 2023-03-31 18:53:49,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288829940, max timestamp=1680288829956Num uniques 859; 2023-03-31 18:53:49,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288829940, max timestamp=1680288829956Num uniques 859; 2023-03-31 18:53:49,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:53:49,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288829940, max timestamp=1680288829956Num uniques 859; 2023-03-31 18:53:49,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:53:49,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:53:49,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288829979, max timestamp=1680288829992Num uniques 1068; 2023-03-31 18:53:49,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-31 18:53:49,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288829979, max timestamp=1680288829992Num uniques 1068; 2023-03-31 18:53:49,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288829979, max timestamp=1680288829992Num uniques 1068; 2023-03-31 18:53:49,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-31 18:53:49,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-31 18:53:50,015 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:50,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:50,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:50,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:50,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:50,019 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:50,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:50,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:50,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:50,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:50,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:50,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:50,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:50,028 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,029 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:50,029 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:50,029 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:50,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680288830022, max timestamp=1680288830033Num uniques 1027; 2023-03-31 18:53:50,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-31 18:53:50,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680288830022, max timestamp=1680288830033Num uniques 1027; 2023-03-31 18:53:50,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-31 18:53:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680288830022, max timestamp=1680288830033Num uniques 1027; 2023-03-31 18:53:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-31 18:53:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:50,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312abed3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:50,039 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:50,039 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,041 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:50,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:50,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:50,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:50,043 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:50,043 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,043 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,043 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:50,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:50,047 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:50,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:50,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:50,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:50,047 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:50,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,050 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,052 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,055 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:50,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:50,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,055 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:50,056 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:50,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:50,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680288830059, max timestamp=1680288830072Num uniques 1136; 2023-03-31 18:53:50,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-03-31 18:53:50,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680288830059, max timestamp=1680288830072Num uniques 1136; 2023-03-31 18:53:50,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-03-31 18:53:50,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680288830059, max timestamp=1680288830072Num uniques 1136; 2023-03-31 18:53:50,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-03-31 18:53:50,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680288830091, max timestamp=1680288830137Num uniques 1095; 2023-03-31 18:53:50,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-03-31 18:53:50,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680288830112, max timestamp=1680288830137Num uniques 1095; 2023-03-31 18:53:50,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680288830091, max timestamp=1680288830137Num uniques 1095; 2023-03-31 18:53:50,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-31 18:53:50,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-03-31 18:53:50,156 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:50,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:50,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:50,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:50,161 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:50,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:50,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:50,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,164 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,164 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:50,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:50,166 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,166 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:50,166 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:50,166 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:50,168 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f595d48[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:50,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:50,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,171 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:50,171 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:50,171 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:50,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:50,173 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:50,176 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:50,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:50,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:50,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:50,177 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:50,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288890181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680288830153, max timestamp=1680288830180Num uniques 1104; 2023-03-31 18:53:50,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-31 18:53:50,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680288830153, max timestamp=1680288830180Num uniques 1204; 2023-03-31 18:53:50,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680288830153, max timestamp=1680288830180Num uniques 1204; 2023-03-31 18:53:50,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-03-31 18:53:50,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-03-31 18:53:50,183 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:50,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:50,186 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:50,187 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:50,187 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:50,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680288830220, max timestamp=1680288830242Num uniques 1113; 2023-03-31 18:53:50,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-03-31 18:53:50,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680288830220, max timestamp=1680288830242Num uniques 1113; 2023-03-31 18:53:50,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680288830220, max timestamp=1680288830242Num uniques 1113; 2023-03-31 18:53:50,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-03-31 18:53:50,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-03-31 18:53:50,287 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:50,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:50,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:50,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:50,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:50,292 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:50,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:50,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:50,293 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,293 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,295 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,295 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:50,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:50,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:50,297 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:50,298 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,298 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:50,298 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:50,298 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:50,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:50,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d306d6[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:50,303 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:50,303 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680288830265, max timestamp=1680288830304Num uniques 1122; 2023-03-31 18:53:50,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680288830265, max timestamp=1680288830304Num uniques 1122; 2023-03-31 18:53:50,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-03-31 18:53:50,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680288830265, max timestamp=1680288830304Num uniques 1122; 2023-03-31 18:53:50,307 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-03-31 18:53:50,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-03-31 18:53:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:50,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:50,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:50,309 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:50,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:50,310 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:50,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:50,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:50,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:50,310 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:50,313 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,313 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,314 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,315 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,321 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,321 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,321 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:50,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:50,321 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,321 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,322 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:50,322 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:50,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680288830330, max timestamp=1680288830343Num uniques 1131; 2023-03-31 18:53:50,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-03-31 18:53:50,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680288830330, max timestamp=1680288830343Num uniques 1131; 2023-03-31 18:53:50,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680288830330, max timestamp=1680288830343Num uniques 1131; 2023-03-31 18:53:50,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-03-31 18:53:50,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-03-31 18:53:50,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/3c470a11e17842e9b1d3d7c81f0717a0 2023-03-31 18:53:50,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680288830347, max timestamp=1680288830387Num uniques 1140; 2023-03-31 18:53:50,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680288830347, max timestamp=1680288830387Num uniques 1140; 2023-03-31 18:53:50,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=17860, suffix heap size=146528,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=390 2023-03-31 18:53:50,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=17860, suffix heap size=146528,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=390 2023-03-31 18:53:50,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680288830347, max timestamp=1680288830387Num uniques 1140; 2023-03-31 18:53:50,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=17860, suffix heap size=146528,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=390 2023-03-31 18:53:50,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/086d5c4031994d42b207f6118b6a0a55 2023-03-31 18:53:50,417 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/cd3d4c3d58cc485e935489f002efad72 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/cd3d4c3d58cc485e935489f002efad72 2023-03-31 18:53:50,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288890416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680288830401, max timestamp=1680288830418Num uniques 1249; 2023-03-31 18:53:50,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=11404, suffix heap size=157864,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=249 2023-03-31 18:53:50,422 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:50,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:50,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680288830401, max timestamp=1680288830418Num uniques 1249; 2023-03-31 18:53:50,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=11404, suffix heap size=157864,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=249 2023-03-31 18:53:50,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680288830401, max timestamp=1680288830418Num uniques 1249; 2023-03-31 18:53:50,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=11404, suffix heap size=157864,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=249 2023-03-31 18:53:50,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:50,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/cd3d4c3d58cc485e935489f002efad72, entries=150, sequenceid=611, filesize=11.9 K 2023-03-31 18:53:50,432 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/3c470a11e17842e9b1d3d7c81f0717a0 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3c470a11e17842e9b1d3d7c81f0717a0 2023-03-31 18:53:50,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:50,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:50,436 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:50,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:50,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:50,437 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,437 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,439 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:50,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,442 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3c470a11e17842e9b1d3d7c81f0717a0, entries=150, sequenceid=611, filesize=11.9 K 2023-03-31 18:53:50,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:50,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:50,443 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,443 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:50,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:50,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,444 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/086d5c4031994d42b207f6118b6a0a55 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/086d5c4031994d42b207f6118b6a0a55 2023-03-31 18:53:50,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:50,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:50,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,445 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:50,445 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:50,445 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,445 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:50,445 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:50,447 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fae425[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:50,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:50,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/086d5c4031994d42b207f6118b6a0a55, entries=150, sequenceid=611, filesize=11.9 K 2023-03-31 18:53:50,456 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=120.76 KB/123660 for 087bf54664c6a240c659c71fb3782134 in 543ms, sequenceid=611, compaction requested=true 2023-03-31 18:53:50,457 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:50,457 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:50,457 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:50,457 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 18:53:50,457 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:53:50,457 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:53:50,457 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,458 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:50,458 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:50,458 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:50,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:50,458 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=127.47 KB heapSize=356.63 KB 2023-03-31 18:53:50,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:50,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:50,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:50,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:50,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:50,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:50,462 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:50,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:50,463 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:50,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:50,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:50,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:50,463 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:50,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,466 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58828 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:53:50,467 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/B is initiating minor compaction (all files) 2023-03-31 18:53:50,467 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/B in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,467 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,466 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58828 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:53:50,467 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/A is initiating minor compaction (all files) 2023-03-31 18:53:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,467 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2255640946054dd1bc8d1b4237795e6f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f84e4521b2244e8f9b6ffb68a6a5d53f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f3d009ec29c54c9f906c9e2bf4c79346, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/4a20eb53f94b44fe82768e40185e86c6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3c470a11e17842e9b1d3d7c81f0717a0] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=57.4 K 2023-03-31 18:53:50,467 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/A in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,468 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/3c78df96cefb47758fdf5e4379aa1d69, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/8a122b3bc14e4649bfbd44cb50590cdc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/87ca19054e6b4be1ac516a1637aac3f4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/bb532030c24d41f1b0300a46ffe8ddf6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/cd3d4c3d58cc485e935489f002efad72] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=57.4 K 2023-03-31 18:53:50,468 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,469 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 3c78df96cefb47758fdf5e4379aa1d69, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1680288826471 2023-03-31 18:53:50,468 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2255640946054dd1bc8d1b4237795e6f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1680288826471 2023-03-31 18:53:50,474 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8a122b3bc14e4649bfbd44cb50590cdc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1680288827114 2023-03-31 18:53:50,475 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f84e4521b2244e8f9b6ffb68a6a5d53f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1680288827114 2023-03-31 18:53:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,475 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 87ca19054e6b4be1ac516a1637aac3f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680288827793 2023-03-31 18:53:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,476 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f3d009ec29c54c9f906c9e2bf4c79346, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680288827793 2023-03-31 18:53:50,476 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,476 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting bb532030c24d41f1b0300a46ffe8ddf6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680288828931 2023-03-31 18:53:50,477 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4a20eb53f94b44fe82768e40185e86c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680288828931 2023-03-31 18:53:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,477 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting cd3d4c3d58cc485e935489f002efad72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1680288829595 2023-03-31 18:53:50,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288890475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,482 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3c470a11e17842e9b1d3d7c81f0717a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1680288829595 2023-03-31 18:53:50,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288890479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288890483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,491 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:50,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:50,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:50,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,492 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:50,492 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:50,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,501 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#B#compaction#45 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:50,502 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#A#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:50,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288890521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,531 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/e7b890bf4a7744b9a912ed420a8f6cb2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/e7b890bf4a7744b9a912ed420a8f6cb2 2023-03-31 18:53:50,533 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/26a6dc476cb84f8d99f13d41ab8464ac as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/26a6dc476cb84f8d99f13d41ab8464ac 2023-03-31 18:53:50,542 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 087bf54664c6a240c659c71fb3782134/A of 087bf54664c6a240c659c71fb3782134 into e7b890bf4a7744b9a912ed420a8f6cb2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:50,542 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:50,542 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/A, priority=11, startTime=1680288830457; duration=0sec 2023-03-31 18:53:50,543 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:50,543 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:53:50,546 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58794 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:53:50,547 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/C is initiating minor compaction (all files) 2023-03-31 18:53:50,547 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/C in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,547 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8a94c494b0b4470bafaf6bb797e2b237, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ec885308c4f5487b93003a43d84d56cd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/9967b5ae49fa4ebaad3882af54dc7225, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/b2645fcfed0a43b3a25ee047389a2736, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/086d5c4031994d42b207f6118b6a0a55] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=57.4 K 2023-03-31 18:53:50,547 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 087bf54664c6a240c659c71fb3782134/B of 087bf54664c6a240c659c71fb3782134 into 26a6dc476cb84f8d99f13d41ab8464ac(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:50,547 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8a94c494b0b4470bafaf6bb797e2b237, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1680288826471 2023-03-31 18:53:50,547 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:50,549 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/B, priority=11, startTime=1680288830457; duration=0sec 2023-03-31 18:53:50,549 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:50,550 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting ec885308c4f5487b93003a43d84d56cd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1680288827114 2023-03-31 18:53:50,550 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9967b5ae49fa4ebaad3882af54dc7225, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680288827793 2023-03-31 18:53:50,551 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting b2645fcfed0a43b3a25ee047389a2736, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680288828931 2023-03-31 18:53:50,551 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 086d5c4031994d42b207f6118b6a0a55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1680288829595 2023-03-31 18:53:50,565 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#C#compaction#47 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:50,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288890584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288890585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,592 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:50,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:50,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288890590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:50,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:50,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:50,599 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:50,599 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,599 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:50,599 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:50,600 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,600 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,602 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,602 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,603 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,603 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:50,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:50,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:50,604 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:50,604 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:50,617 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63421e2f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:50,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:50,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,631 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:50,632 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,632 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:50,633 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,633 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:50,634 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,634 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:50,637 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:50,637 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:50,637 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:50,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:50,638 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:50,641 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,641 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,641 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,644 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,649 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:50,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:50,651 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,652 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:50,652 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:50,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288890727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,752 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:50,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:50,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:50,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:50,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:50,768 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:50,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:50,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:50,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,771 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:50,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:50,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:50,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:50,774 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,774 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:50,774 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:50,774 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:50,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:50,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:50,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:50,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f4bc12[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:50,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:50,785 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:50,788 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288890787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,788 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:50,789 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,790 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:50,790 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,790 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:50,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288890788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:50,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:50,795 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:50,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:50,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:50,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:50,795 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:50,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,798 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288890797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,799 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:50,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:50,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:50,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,802 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:50,802 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:50,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,803 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=817 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/11e1297e94d745ef8afefd26cb5826e8 2023-03-31 18:53:50,903 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:50,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:50,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:50,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:50,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:50,909 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:50,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:50,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:50,910 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,910 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:50,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:50,916 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:50,916 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:50,916 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:50,917 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:50,917 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:50,917 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:50,917 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:50,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5daae3ac[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:50,930 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:50,930 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,931 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:50,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:50,937 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,937 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,937 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:50,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:50,947 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:50,947 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:50,947 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:50,947 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:50,947 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:50,949 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,949 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,950 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:50,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:50,955 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:50,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:50,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:50,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:50,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:50,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:50,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:50,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,956 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:50,956 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:50,956 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:50,956 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:50,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:50,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:50,993 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/780fa56fc9fe4b4ead088ea0e2bd5359 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/780fa56fc9fe4b4ead088ea0e2bd5359 2023-03-31 18:53:51,003 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 087bf54664c6a240c659c71fb3782134/C of 087bf54664c6a240c659c71fb3782134 into 780fa56fc9fe4b4ead088ea0e2bd5359(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:51,003 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:51,003 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/C, priority=11, startTime=1680288830457; duration=0sec 2023-03-31 18:53:51,004 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:51,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:51,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288891030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,056 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:51,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:51,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:51,063 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:51,063 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,063 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:51,063 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:51,063 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,063 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:51,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:51,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:51,067 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,067 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:51,067 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:51,067 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:51,078 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,078 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da1eff0[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:51,080 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:51,081 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,083 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:51,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:51,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:51,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:51,089 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:51,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:51,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:51,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:51,089 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:51,091 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,091 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,091 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:51,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288891092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,093 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,094 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:51,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288891094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,099 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,099 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,099 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,099 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,100 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,100 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,100 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,100 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,100 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,100 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:51,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:51,100 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:51,100 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:51,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:51,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288891101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,201 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:51,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:51,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:51,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:51,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:51,205 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:51,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:51,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:51,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,208 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,209 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,209 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,209 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:51,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:51,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:51,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:51,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:51,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:51,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:51,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:51,211 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,211 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:51,211 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:51,211 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:51,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:51,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:51,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:51,213 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,213 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,213 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:51,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e613b0d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:51,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:51,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:51,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:51,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:51,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:51,221 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:51,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:51,231 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:51,231 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:51,231 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:51,231 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:51,231 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:51,234 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,234 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,234 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,235 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,237 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,241 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,241 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,241 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,242 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:51,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:51,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:51,242 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:51,242 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:51,246 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,246 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=817 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/45c84078f8bd47f4929b7f4c949e112c 2023-03-31 18:53:51,342 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:51,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:51,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:51,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:51,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:51,346 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:51,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:51,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:51,347 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,347 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,348 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,349 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,350 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,350 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:51,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:51,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:51,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:51,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:51,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:51,358 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:51,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3db60cff[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:51,360 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:51,361 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:51,362 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,362 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:51,364 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:51,364 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,364 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:51,371 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:51,375 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:51,375 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:51,375 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:51,375 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:51,375 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:51,377 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,377 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,378 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,377 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,381 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,387 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,387 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,387 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,387 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,387 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:51,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:51,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,388 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:51,388 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:51,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:51,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:51,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60024 deadline: 1680288891477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,481 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:53:51,489 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:51,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:51,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:51,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:51,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:51,493 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:51,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:51,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:51,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,495 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,495 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,496 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,496 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:51,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:51,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:51,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:51,499 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@378b2f66[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:51,502 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:51,502 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:51,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:51,509 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:51,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:51,512 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:51,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:51,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:51,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:51,513 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:51,514 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,514 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,514 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,515 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,516 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,519 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,519 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,519 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,519 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,519 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,519 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,519 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,519 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,519 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:51,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:51,519 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,520 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:51,520 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:51,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:51,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:51,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60006 deadline: 1680288891536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:51,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60008 deadline: 1680288891597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:51,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288891599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:51,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288891606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,620 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:51,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:51,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:51,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:51,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:51,624 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:51,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:51,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:51,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,628 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,628 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,629 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,629 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:51,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:51,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:51,631 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,631 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:51,631 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:51,631 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:51,635 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,635 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,635 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:51,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35579808[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:51,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:51,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,639 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:51,639 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:51,639 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:51,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:51,641 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:51,645 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:51,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:51,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:51,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:51,645 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:51,646 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,646 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,648 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,650 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,656 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:51,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:51,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,657 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,657 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,657 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,657 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:51,657 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:51,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=817 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/eb289c72dc354401accc57c8bcd73260 2023-03-31 18:53:51,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/11e1297e94d745ef8afefd26cb5826e8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/11e1297e94d745ef8afefd26cb5826e8 2023-03-31 18:53:51,757 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:51,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:51,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:51,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:51,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:51,763 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:51,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:51,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:51,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/11e1297e94d745ef8afefd26cb5826e8, entries=200, sequenceid=817, filesize=14.3 K 2023-03-31 18:53:51,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/45c84078f8bd47f4929b7f4c949e112c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/45c84078f8bd47f4929b7f4c949e112c 2023-03-31 18:53:51,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:51,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:51,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:51,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:51,771 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,771 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:51,771 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:51,771 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:51,773 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/45c84078f8bd47f4929b7f4c949e112c, entries=150, sequenceid=817, filesize=11.9 K 2023-03-31 18:53:51,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/eb289c72dc354401accc57c8bcd73260 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eb289c72dc354401accc57c8bcd73260 2023-03-31 18:53:51,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:51,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429974fc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:51,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:51,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:51,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:51,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:51,781 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,781 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:51,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,783 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,783 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:51,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:51,788 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:51,788 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:51,788 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:51,788 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:51,788 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:51,789 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eb289c72dc354401accc57c8bcd73260, entries=150, sequenceid=817, filesize=11.9 K 2023-03-31 18:53:51,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:51,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~356.58 KB/365136, currentSize=53.67 KB/54960 for 087bf54664c6a240c659c71fb3782134 in 1332ms, sequenceid=817, compaction requested=false 2023-03-31 18:53:51,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:51,791 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:51,793 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:51,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:51,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:51,800 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,801 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,801 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:51,801 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:51,801 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:51,801 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:51,901 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:51,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:51,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:51,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:51,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:51,906 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:51,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:51,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:51,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,908 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:51,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:51,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:51,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:51,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:51,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:51,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:51,911 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:51,911 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:51,911 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:51,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:51,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:51,912 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:51,912 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:51,912 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:53:51,912 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:51,913 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:51,913 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:51,913 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:51,913 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:51,913 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:51,925 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/9f866d746ebe49609a6429b27972c337 2023-03-31 18:53:51,954 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/f788252a649d4789b060bc17f8147fa9 2023-03-31 18:53:52,382 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/ee3fd02fbd324d63a599b7301fa0c157 2023-03-31 18:53:52,394 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/9f866d746ebe49609a6429b27972c337 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9f866d746ebe49609a6429b27972c337 2023-03-31 18:53:52,410 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9f866d746ebe49609a6429b27972c337, entries=100, sequenceid=831, filesize=9.5 K 2023-03-31 18:53:52,412 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/f788252a649d4789b060bc17f8147fa9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f788252a649d4789b060bc17f8147fa9 2023-03-31 18:53:52,423 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f788252a649d4789b060bc17f8147fa9, entries=100, sequenceid=831, filesize=9.5 K 2023-03-31 18:53:52,433 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/ee3fd02fbd324d63a599b7301fa0c157 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ee3fd02fbd324d63a599b7301fa0c157 2023-03-31 18:53:52,442 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ee3fd02fbd324d63a599b7301fa0c157, entries=100, sequenceid=831, filesize=9.5 K 2023-03-31 18:53:52,444 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 087bf54664c6a240c659c71fb3782134 in 532ms, sequenceid=831, compaction requested=true 2023-03-31 18:53:52,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:52,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:52,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:52,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:52,452 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4559a4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:52,455 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:52,455 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,457 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:52,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:52,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:52,458 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,458 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:52,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:52,460 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:52,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:52,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:52,465 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:52,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:52,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:52,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:52,465 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:52,467 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,468 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:52,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,468 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,469 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:52,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:52,475 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:52,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:52,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:52,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:52,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,476 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,476 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,476 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,476 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:52,477 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:52,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:52,477 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:52,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:52,571 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:53:52,571 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:52,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:52,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:52,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:52,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:52,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:52,577 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:52,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:52,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:52,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:52,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:52,589 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:52,589 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:52,589 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:52,589 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:52,589 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,589 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,591 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:52,591 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:52,591 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,591 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:52,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:52,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:52,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:52,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:52,608 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:52,608 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:52,608 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:52,608 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:52,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:52,613 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,613 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,613 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,614 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21dbbf35[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:52,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:52,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,618 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:52,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:52,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:52,619 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,619 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:52,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:52,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:52,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:52,620 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:52,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:52,624 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:52,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:52,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:52,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:52,624 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:52,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288832595, max timestamp=1680288832623Num uniques 859; 2023-03-31 18:53:52,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:53:52,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288832595, max timestamp=1680288832623Num uniques 859; 2023-03-31 18:53:52,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:53:52,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288832595, max timestamp=1680288832623Num uniques 859; 2023-03-31 18:53:52,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:53:52,630 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,630 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:52,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,631 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,633 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,634 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,638 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:52,638 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:52,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:52,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:52,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:52,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:52,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:52,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,640 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,640 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:52,640 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:52,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288892649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60008 deadline: 1680288892649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288892654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288892655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,740 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:52,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:52,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:52,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:52,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:52,744 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:52,744 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:52,744 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:52,744 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:52,744 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,744 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,747 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:52,747 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:52,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:52,747 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,747 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:52,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:52,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:52,749 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:52,749 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:52,749 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:52,749 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:52,749 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:52,749 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:52,749 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:52,749 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:52,749 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:52,753 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288892756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60008 deadline: 1680288892756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288892758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288892759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:52,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28c5f4de[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:52,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:52,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:52,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:52,772 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:52,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:52,777 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:52,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:52,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:52,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:52,777 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:52,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,785 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,799 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:52,799 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:52,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:52,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:52,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:52,799 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:52,799 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:52,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,799 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:52,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:52,900 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:52,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:52,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:52,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:52,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:52,908 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:52,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:52,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:52,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:52,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,916 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:52,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:52,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:52,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:52,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:52,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:52,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:52,921 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:52,921 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:52,921 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:52,921 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:52,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:52,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:52,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:52,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:52,923 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,925 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:52,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21724bae[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:52,928 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:52,928 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:52,929 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,929 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:52,931 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:52,931 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:52,931 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:52,931 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:52,934 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:52,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:52,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:52,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:52,934 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:52,937 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:52,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:52,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,938 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,939 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:52,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:52,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:52,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:52,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:52,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:52,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:52,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:52,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:52,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:52,943 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:52,943 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:52,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:52,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288892961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288892961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60008 deadline: 1680288892962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:52,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:52,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288892962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,057 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:53,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:53,058 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=841 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/83c6304bfd574d2e921a68b7db8b9769 2023-03-31 18:53:53,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:53,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:53,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:53,061 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:53,061 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,061 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:53,061 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:53,062 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,062 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,064 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:53,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:53,068 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,068 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:53,068 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:53,068 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:53,073 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:53,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449fbe5d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:53,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:53,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,077 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,077 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:53,077 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:53,077 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:53,078 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,078 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:53,079 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,079 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:53,079 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,079 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:53,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:53,082 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:53,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:53,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:53,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:53,082 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:53,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,085 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,086 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,089 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,089 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,089 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,089 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:53,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:53,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,089 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,089 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,089 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,089 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,089 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,089 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:53,089 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,189 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:53,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:53,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:53,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:53,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:53,193 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:53,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:53,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:53,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,195 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:53,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:53,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:53,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:53,197 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,197 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:53,197 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:53,197 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:53,199 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,199 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,199 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,199 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:53,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@384183ec[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:53,201 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:53,201 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:53,204 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,204 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:53,205 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,205 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:53,208 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:53,208 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:53,208 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:53,208 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:53,208 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:53,210 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,210 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,210 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,210 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,212 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:53,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:53,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,215 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,215 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,215 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,215 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:53,215 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:53,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288893267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288893268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288893268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60008 deadline: 1680288893274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,315 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:53,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:53,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:53,320 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:53,321 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,321 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:53,321 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:53,321 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,321 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,326 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,326 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,328 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,328 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:53,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:53,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:53,335 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:53,335 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,335 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:53,339 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,339 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:53,339 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:53,339 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:53,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa5f33e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:53,344 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:53,344 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,352 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:53,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:53,354 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:53,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:53,360 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:53,360 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:53,360 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:53,360 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:53,360 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:53,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,370 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,373 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:53,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:53,375 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:53,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,375 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:53,376 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:53,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,476 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:53,476 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=841 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/ded953bb6cf240ffbe58425b9676ae09 2023-03-31 18:53:53,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:53,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:53,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:53,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:53,479 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:53,480 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,480 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:53,480 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:53,480 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,480 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,482 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,482 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,483 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,483 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:53,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:53,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:53,484 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:53,484 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,484 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:53,484 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:53,484 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:53,484 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,484 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:53,484 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:53,484 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:53,486 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:53,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c39e05f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:53,490 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:53,490 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,493 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,493 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:53,494 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:53,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:53,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:53,495 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=841 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/d9179821e57f448c95eac24794299919 2023-03-31 18:53:53,499 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:53,499 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,499 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,499 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/83c6304bfd574d2e921a68b7db8b9769 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/83c6304bfd574d2e921a68b7db8b9769 2023-03-31 18:53:53,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:53,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:53,506 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:53,507 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:53,507 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:53,507 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:53,507 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:53,509 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,509 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,509 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/83c6304bfd574d2e921a68b7db8b9769, entries=150, sequenceid=841, filesize=11.9 K 2023-03-31 18:53:53,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/ded953bb6cf240ffbe58425b9676ae09 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/ded953bb6cf240ffbe58425b9676ae09 2023-03-31 18:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/ded953bb6cf240ffbe58425b9676ae09, entries=100, sequenceid=841, filesize=9.5 K 2023-03-31 18:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,517 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/d9179821e57f448c95eac24794299919 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d9179821e57f448c95eac24794299919 2023-03-31 18:53:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,529 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d9179821e57f448c95eac24794299919, entries=100, sequenceid=841, filesize=9.5 K 2023-03-31 18:53:53,532 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 087bf54664c6a240c659c71fb3782134 in 961ms, sequenceid=841, compaction requested=true 2023-03-31 18:53:53,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:53,533 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:53,533 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:53:53,533 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:53:53,533 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:53,533 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:53,535 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,535 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,535 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,535 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,535 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:53,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:53,535 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,535 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44368 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:53:53,536 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,536 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:53,536 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:53,536 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49248 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:53:53,536 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/A is initiating minor compaction (all files) 2023-03-31 18:53:53,536 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/A in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,536 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/e7b890bf4a7744b9a912ed420a8f6cb2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/11e1297e94d745ef8afefd26cb5826e8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9f866d746ebe49609a6429b27972c337, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/83c6304bfd574d2e921a68b7db8b9769] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=48.1 K 2023-03-31 18:53:53,536 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,536 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/B is initiating minor compaction (all files) 2023-03-31 18:53:53,537 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/B in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,537 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/26a6dc476cb84f8d99f13d41ab8464ac, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/45c84078f8bd47f4929b7f4c949e112c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f788252a649d4789b060bc17f8147fa9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/ded953bb6cf240ffbe58425b9676ae09] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=43.3 K 2023-03-31 18:53:53,537 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting e7b890bf4a7744b9a912ed420a8f6cb2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1680288829595 2023-03-31 18:53:53,538 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 26a6dc476cb84f8d99f13d41ab8464ac, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1680288829595 2023-03-31 18:53:53,538 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 11e1297e94d745ef8afefd26cb5826e8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=817, earliestPutTs=1680288830444 2023-03-31 18:53:53,538 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 45c84078f8bd47f4929b7f4c949e112c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=817, earliestPutTs=1680288830444 2023-03-31 18:53:53,539 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9f866d746ebe49609a6429b27972c337, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680288830478 2023-03-31 18:53:53,539 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f788252a649d4789b060bc17f8147fa9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680288830478 2023-03-31 18:53:53,539 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 83c6304bfd574d2e921a68b7db8b9769, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1680288832559 2023-03-31 18:53:53,539 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,539 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ded953bb6cf240ffbe58425b9676ae09, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1680288832559 2023-03-31 18:53:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,539 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,565 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#A#compaction#56 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:53,569 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#B#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:53,593 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/70dff6fb8d974ff8be84946e0ce5d6d4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/70dff6fb8d974ff8be84946e0ce5d6d4 2023-03-31 18:53:53,595 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/af3031bbcd2540be84bf0700fd546cb3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/af3031bbcd2540be84bf0700fd546cb3 2023-03-31 18:53:53,600 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 087bf54664c6a240c659c71fb3782134/A of 087bf54664c6a240c659c71fb3782134 into 70dff6fb8d974ff8be84946e0ce5d6d4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:53,600 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:53,600 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/A, priority=12, startTime=1680288833533; duration=0sec 2023-03-31 18:53:53,600 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:53,600 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:53:53,607 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44334 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:53:53,607 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/C is initiating minor compaction (all files) 2023-03-31 18:53:53,607 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/C in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,607 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/780fa56fc9fe4b4ead088ea0e2bd5359, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eb289c72dc354401accc57c8bcd73260, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ee3fd02fbd324d63a599b7301fa0c157, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d9179821e57f448c95eac24794299919] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=43.3 K 2023-03-31 18:53:53,608 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 780fa56fc9fe4b4ead088ea0e2bd5359, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1680288829595 2023-03-31 18:53:53,609 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting eb289c72dc354401accc57c8bcd73260, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=817, earliestPutTs=1680288830444 2023-03-31 18:53:53,609 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 087bf54664c6a240c659c71fb3782134/B of 087bf54664c6a240c659c71fb3782134 into af3031bbcd2540be84bf0700fd546cb3(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:53,610 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:53,610 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/B, priority=12, startTime=1680288833533; duration=0sec 2023-03-31 18:53:53,610 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting ee3fd02fbd324d63a599b7301fa0c157, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680288830478 2023-03-31 18:53:53,610 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:53,610 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d9179821e57f448c95eac24794299919, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1680288832559 2023-03-31 18:53:53,629 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#C#compaction#58 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:53,636 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:53,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:53,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:53,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:53,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:53,641 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:53,641 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,641 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:53,642 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:53,642 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,642 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,643 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,644 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,644 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,644 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:53,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:53,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:53,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:53,657 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:53,657 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:53,664 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,664 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:53,664 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-31 18:53:53,664 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:53,665 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:53,666 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:53,666 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:53,666 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:53,667 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:53,707 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/2d443de1f30f43d59049682c0008d46c 2023-03-31 18:53:53,727 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/b7a22742978c4f6d927db02b69b5c22d 2023-03-31 18:53:53,758 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/7a02aad6ac5d49f3b389d23c75e51cd7 2023-03-31 18:53:53,765 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/2d443de1f30f43d59049682c0008d46c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2d443de1f30f43d59049682c0008d46c 2023-03-31 18:53:53,770 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2d443de1f30f43d59049682c0008d46c, entries=150, sequenceid=879, filesize=11.9 K 2023-03-31 18:53:53,771 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/b7a22742978c4f6d927db02b69b5c22d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b7a22742978c4f6d927db02b69b5c22d 2023-03-31 18:53:53,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:53,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:53,780 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b7a22742978c4f6d927db02b69b5c22d, entries=150, sequenceid=879, filesize=11.9 K 2023-03-31 18:53:53,781 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/7a02aad6ac5d49f3b389d23c75e51cd7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/7a02aad6ac5d49f3b389d23c75e51cd7 2023-03-31 18:53:53,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60008 deadline: 1680288893778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288893779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288893782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288893783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,788 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/7a02aad6ac5d49f3b389d23c75e51cd7, entries=150, sequenceid=879, filesize=11.9 K 2023-03-31 18:53:53,789 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=46.96 KB/48090 for 087bf54664c6a240c659c71fb3782134 in 125ms, sequenceid=879, compaction requested=false 2023-03-31 18:53:53,789 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:53,789 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:53,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35edcad3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:53,793 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:53,793 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,795 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:53,795 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:53,795 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:53,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:53,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,798 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:53,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:53,801 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:53,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:53,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:53,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:53,801 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:53,804 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,804 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,804 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,809 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,813 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:53,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:53,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:53,814 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:53,814 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:53,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:53,887 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:53:53,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:53,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:53,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:53,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:53,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:53,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:53,914 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:53,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:53,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:53,920 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:53,920 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,920 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:53,920 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:53,920 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,921 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:53,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:53,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:53,924 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,924 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:53,924 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:53,924 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:53,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1077161[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:53,927 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:53,927 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:53,929 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,929 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:53,936 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:53,936 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:53,936 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:53,937 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:53,937 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:53,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:53,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,939 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,941 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:53,943 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:53,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:53,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:53,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:53,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:53,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:53,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:53,943 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:53,943 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:53,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,944 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:53,944 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:53,944 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:53,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288893943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680288833908, max timestamp=1680288833946Num uniques 859; 2023-03-31 18:53:53,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-31 18:53:53,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680288833908, max timestamp=1680288833946Num uniques 859; 2023-03-31 18:53:53,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-31 18:53:53,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680288833908, max timestamp=1680288833946Num uniques 859; 2023-03-31 18:53:53,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-31 18:53:53,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288893971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:53,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:53,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288893971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,044 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:54,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:54,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:54,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:54,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:54,047 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:54,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:54,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:54,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288894049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,052 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:54,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:54,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:54,055 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,055 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:54,055 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:54,055 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:54,060 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/db4f92f213a34a279b4ef7db155ef910 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/db4f92f213a34a279b4ef7db155ef910 2023-03-31 18:53:54,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:54,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@112b5e7c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:54,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:54,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:54,066 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,066 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:54,067 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 087bf54664c6a240c659c71fb3782134/C of 087bf54664c6a240c659c71fb3782134 into db4f92f213a34a279b4ef7db155ef910(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:54,067 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:54,067 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/C, priority=12, startTime=1680288833533; duration=0sec 2023-03-31 18:53:54,067 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,068 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:54,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:54,072 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:54,072 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:54,072 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:54,072 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:54,072 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:54,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288894074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:54,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288894077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,080 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:54,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:54,085 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,085 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:54,086 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:54,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,186 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:54,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:54,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:54,196 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:54,196 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,196 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:54,196 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:54,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,198 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,199 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,199 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,199 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,199 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:54,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:54,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:54,201 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:54,201 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,201 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:54,214 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,214 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:54,214 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:54,214 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:54,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,216 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:54,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52536f1e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:54,219 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:54,219 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,221 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,221 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:54,221 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:54,222 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:54,222 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,222 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:54,223 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:54,223 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,223 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:54,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:54,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:54,227 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:54,227 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:54,227 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:54,227 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:54,227 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:54,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,231 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,233 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,237 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:54,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:54,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:54,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,237 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,238 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,238 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,238 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,238 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,237 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:54,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:54,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288894255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:54,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288894279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288894280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,314 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=890 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/84ee81901db442e690dc23f87c50375e 2023-03-31 18:53:54,338 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:54,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:54,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:54,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:54,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:54,342 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:54,342 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,342 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:54,342 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:54,342 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,342 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,344 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,344 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,345 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,345 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:54,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:54,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:54,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:54,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:54,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:54,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:54,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:54,346 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,347 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:54,347 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:54,347 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,347 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:54,347 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:54,347 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,347 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:54,347 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:54,348 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:54,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b39cd5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:54,350 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:54,350 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,351 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:54,351 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,351 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:54,352 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,353 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,353 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,353 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:54,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:54,355 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:54,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:54,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:54,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:54,355 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:54,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,363 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,364 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:54,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:54,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,369 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:54,370 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:54,370 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:54,470 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:54,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:54,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:54,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:54,473 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:54,473 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,473 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:54,473 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:54,474 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,474 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,476 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,476 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,476 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,477 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:54,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:54,480 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,480 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:54,480 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:54,480 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:54,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:54,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:54,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,486 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@338578fb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:54,489 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:54,489 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:54,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:54,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,494 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:54,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:54,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:54,494 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:54,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:54,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:54,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:54,494 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:54,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,498 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,500 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,503 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:54,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:54,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:54,503 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:54,504 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:54,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:54,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288894559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:54,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288894581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288894587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,604 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:54,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:54,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:54,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:54,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:54,607 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:54,607 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,607 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:54,607 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:54,607 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,607 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,609 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,609 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,609 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,609 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:54,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:54,611 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,611 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:54,611 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:54,611 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:54,612 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78961849[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:54,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:54,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,618 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:54,618 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,618 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:54,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:54,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:54,619 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:54,622 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:54,623 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:54,623 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:54,623 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:54,623 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:54,628 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,628 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,628 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,629 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,633 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,638 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,640 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:54,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:54,640 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:54,640 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:54,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=890 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/19aac34c89fc479d8e0dda0d7f71173d 2023-03-31 18:53:54,741 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:54,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:54,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:54,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:54,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:54,744 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:54,744 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,744 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:54,744 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:54,745 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,745 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,746 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,746 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,747 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,747 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:54,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:54,748 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,748 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:54,748 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:54,748 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:54,750 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:54,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f4b3da[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:54,752 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:54,752 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,753 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,753 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:54,753 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:54,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:54,753 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:54,755 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,755 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:54,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,755 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,755 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:54,758 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:54,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:54,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:54,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:54,758 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:54,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,760 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,762 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:54,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:54,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:54,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,768 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:54,768 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:54,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:54,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60008 deadline: 1680288894790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,869 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:54,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:54,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:54,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:54,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:54,872 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:54,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:54,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:54,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,875 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,875 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,875 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,875 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:54,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:54,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:54,877 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,877 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:54,877 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:54,877 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:54,879 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,879 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,879 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,879 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24b4c405[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:54,881 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:54,881 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,882 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:54,882 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,882 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:54,883 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,883 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,883 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,883 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:54,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:54,886 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:54,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:54,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:54,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:54,886 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:54,888 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,888 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:54,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,888 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:54,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,889 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,890 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,892 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,893 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,893 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,893 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:54,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:54,893 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:54,893 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:54,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,893 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:54,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:54,893 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,893 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:54,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,893 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:54,893 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:54,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:54,993 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:54,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:54,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:54,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:54,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:54,996 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:54,996 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:54,997 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:54,997 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:54,997 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:54,997 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,998 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,998 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:54,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:54,999 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:54,999 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:54,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:55,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:55,004 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:55,004 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:55,004 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:55,004 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:55,004 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:55,005 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:55,006 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,007 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,007 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:55,009 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cee32c8[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:55,009 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,011 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,011 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:55,011 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,011 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:55,011 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:55,011 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:55,013 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:55,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:55,016 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:55,016 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:55,016 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:55,016 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:55,016 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:55,018 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,018 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,018 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:55,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:55,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:55,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:55,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:55,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:55,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:55,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:55,023 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:55,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:55,023 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:55,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:55,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:55,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288895063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:55,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288895085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:55,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288895094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,123 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:55,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:55,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:55,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:55,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:55,128 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:55,128 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:55,128 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:55,128 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:55,128 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,128 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,130 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:55,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:55,130 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,131 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,131 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:55,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:55,138 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f2214c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:55,140 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:55,140 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:55,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:55,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:55,143 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,143 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:55,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=890 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/2b1470cdf7774b9d84325b07f5081ee0 2023-03-31 18:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:55,147 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/84ee81901db442e690dc23f87c50375e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84ee81901db442e690dc23f87c50375e 2023-03-31 18:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:55,155 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:55,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:55,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:55,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:55,155 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:55,156 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84ee81901db442e690dc23f87c50375e, entries=150, sequenceid=890, filesize=11.9 K 2023-03-31 18:53:55,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,158 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/19aac34c89fc479d8e0dda0d7f71173d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/19aac34c89fc479d8e0dda0d7f71173d 2023-03-31 18:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:55,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:55,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:55,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:55,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:55,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:55,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:55,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:55,163 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:55,163 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:55,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/19aac34c89fc479d8e0dda0d7f71173d, entries=150, sequenceid=890, filesize=11.9 K 2023-03-31 18:53:55,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/2b1470cdf7774b9d84325b07f5081ee0 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/2b1470cdf7774b9d84325b07f5081ee0 2023-03-31 18:53:55,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/2b1470cdf7774b9d84325b07f5081ee0, entries=150, sequenceid=890, filesize=11.9 K 2023-03-31 18:53:55,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 087bf54664c6a240c659c71fb3782134 in 1287ms, sequenceid=890, compaction requested=true 2023-03-31 18:53:55,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:55,175 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:55,175 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:55,175 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:55,175 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:55,175 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:55,178 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:55,178 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:55,178 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/A is initiating minor compaction (all files) 2023-03-31 18:53:55,178 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/B is initiating minor compaction (all files) 2023-03-31 18:53:55,178 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/A in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,178 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/B in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,178 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/70dff6fb8d974ff8be84946e0ce5d6d4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2d443de1f30f43d59049682c0008d46c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84ee81901db442e690dc23f87c50375e] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=36.3 K 2023-03-31 18:53:55,178 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/af3031bbcd2540be84bf0700fd546cb3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b7a22742978c4f6d927db02b69b5c22d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/19aac34c89fc479d8e0dda0d7f71173d] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=36.3 K 2023-03-31 18:53:55,179 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 70dff6fb8d974ff8be84946e0ce5d6d4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1680288830479 2023-03-31 18:53:55,179 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting af3031bbcd2540be84bf0700fd546cb3, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1680288830479 2023-03-31 18:53:55,179 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b7a22742978c4f6d927db02b69b5c22d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1680288832642 2023-03-31 18:53:55,179 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 2d443de1f30f43d59049682c0008d46c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1680288832642 2023-03-31 18:53:55,180 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 84ee81901db442e690dc23f87c50375e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=890, earliestPutTs=1680288833781 2023-03-31 18:53:55,180 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 19aac34c89fc479d8e0dda0d7f71173d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=890, earliestPutTs=1680288833781 2023-03-31 18:53:55,198 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#A#compaction#65 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:55,201 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#B#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:55,220 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/98e62a18dbf7414b961ed3daf771373d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/98e62a18dbf7414b961ed3daf771373d 2023-03-31 18:53:55,230 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/B of 087bf54664c6a240c659c71fb3782134 into 98e62a18dbf7414b961ed3daf771373d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:55,230 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:55,230 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/B, priority=13, startTime=1680288835175; duration=0sec 2023-03-31 18:53:55,230 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:55,230 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:55,232 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:55,232 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/C is initiating minor compaction (all files) 2023-03-31 18:53:55,232 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/C in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,232 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/db4f92f213a34a279b4ef7db155ef910, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/7a02aad6ac5d49f3b389d23c75e51cd7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/2b1470cdf7774b9d84325b07f5081ee0] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=36.3 K 2023-03-31 18:53:55,233 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting db4f92f213a34a279b4ef7db155ef910, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1680288830479 2023-03-31 18:53:55,233 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 7a02aad6ac5d49f3b389d23c75e51cd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1680288832642 2023-03-31 18:53:55,234 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2b1470cdf7774b9d84325b07f5081ee0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=890, earliestPutTs=1680288833781 2023-03-31 18:53:55,244 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#C#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:55,263 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:55,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:55,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:55,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:55,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:55,267 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:55,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:55,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:55,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:55,268 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,268 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,277 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:55,277 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:55,277 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,277 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:55,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:55,279 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,279 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:55,279 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-03-31 18:53:55,280 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:55,280 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:55,280 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:55,280 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:55,280 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:55,280 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:55,291 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=925 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/00f07a1684754f1dae5cbdcd0ede3f98 2023-03-31 18:53:55,613 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/d8b8e3a4848648798d9c3eca8fdf3a6e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/d8b8e3a4848648798d9c3eca8fdf3a6e 2023-03-31 18:53:55,619 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/A of 087bf54664c6a240c659c71fb3782134 into d8b8e3a4848648798d9c3eca8fdf3a6e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:55,620 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:55,620 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/A, priority=13, startTime=1680288835175; duration=0sec 2023-03-31 18:53:55,620 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:55,668 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/c2fdef4056a044fbae8e55e26f9d42c5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/c2fdef4056a044fbae8e55e26f9d42c5 2023-03-31 18:53:55,676 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/C of 087bf54664c6a240c659c71fb3782134 into c2fdef4056a044fbae8e55e26f9d42c5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:55,676 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:55,676 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/C, priority=13, startTime=1680288835175; duration=0sec 2023-03-31 18:53:55,677 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:55,704 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=925 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/86266f771f9d473abd2dd9d7068dd23d 2023-03-31 18:53:55,725 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=925 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/341488f66c914264a0b0c7db85ccddc9 2023-03-31 18:53:55,731 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/00f07a1684754f1dae5cbdcd0ede3f98 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/00f07a1684754f1dae5cbdcd0ede3f98 2023-03-31 18:53:55,736 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/00f07a1684754f1dae5cbdcd0ede3f98, entries=150, sequenceid=925, filesize=11.9 K 2023-03-31 18:53:55,738 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/86266f771f9d473abd2dd9d7068dd23d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/86266f771f9d473abd2dd9d7068dd23d 2023-03-31 18:53:55,744 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/86266f771f9d473abd2dd9d7068dd23d, entries=150, sequenceid=925, filesize=11.9 K 2023-03-31 18:53:55,745 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/341488f66c914264a0b0c7db85ccddc9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/341488f66c914264a0b0c7db85ccddc9 2023-03-31 18:53:55,750 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/341488f66c914264a0b0c7db85ccddc9, entries=150, sequenceid=925, filesize=11.9 K 2023-03-31 18:53:55,751 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=0 B/0 for 087bf54664c6a240c659c71fb3782134 in 472ms, sequenceid=925, compaction requested=false 2023-03-31 18:53:55,751 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:55,751 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:55,753 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,753 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:55,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@608c366b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:55,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:55,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,757 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,757 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:55,757 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:55,757 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:55,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:55,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:55,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:55,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:55,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:55,767 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:55,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:55,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:55,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:55,767 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:55,769 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,769 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,769 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:55,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:55,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:55,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:55,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:55,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:55,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,774 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:55,774 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:55,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:55,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,774 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:55,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:55,874 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:55,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:55,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:55,880 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:55,880 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:55,880 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:55,880 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:55,881 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,881 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,882 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:55,882 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,883 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,883 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:55,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:55,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:55,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:55,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:55,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:55,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:55,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:55,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:55,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,897 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:55,897 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:55,897 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:55,897 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:55,897 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:55,897 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,897 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:55,897 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:55,899 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,899 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,899 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,899 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:55,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@621a5293[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:55,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:55,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,903 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:55,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:55,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:55,904 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,904 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:55,906 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:55,906 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:55,906 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:55,906 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:55,912 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:55,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:55,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:55,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:55,912 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:55,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:55,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,915 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:55,916 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:55,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:55,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:55,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:55,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:55,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:55,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:55,920 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:55,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:55,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:55,921 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,921 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:55,921 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:55,921 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:55,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:55,921 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:56,022 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:56,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:56,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:56,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:56,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:56,025 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:56,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:56,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:56,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:56,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:56,045 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,045 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:56,045 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:56,045 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:56,046 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,046 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2faf446a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:56,049 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:56,049 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:56,050 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,050 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:56,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:56,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:56,054 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:56,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:56,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:56,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:56,054 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:56,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,057 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:56,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:56,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,060 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:56,061 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:56,061 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:56,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:56,077 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:53:56,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:56,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:56,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:56,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:56,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:56,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:56,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288836107, max timestamp=1680288836112Num uniques 859; 2023-03-31 18:53:56,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288836107, max timestamp=1680288836112Num uniques 859; 2023-03-31 18:53:56,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:53:56,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:53:56,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288836107, max timestamp=1680288836112Num uniques 859; 2023-03-31 18:53:56,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:53:56,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288896138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288896139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288896140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:56,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:56,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:56,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:56,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:56,165 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:56,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:56,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:56,166 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,166 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:56,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:56,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:56,173 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,173 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:56,174 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:56,174 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:56,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:56,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a32f892[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:56,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:56,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:56,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:56,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:56,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:56,185 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:56,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:56,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:56,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:56,185 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:56,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,187 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,189 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:56,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:56,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,197 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:56,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,197 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:56,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288896241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288896242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288896244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,297 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:56,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:56,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:56,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:56,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:56,302 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:56,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:56,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:56,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,303 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,304 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,304 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:56,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:56,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:56,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:56,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:56,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:56,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8ca0d1[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:56,312 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:56,312 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,315 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,315 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:56,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:56,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:56,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:56,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:56,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,321 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:56,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:56,325 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:56,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:56,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:56,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:56,325 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:56,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,327 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,328 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,331 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:56,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:56,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,331 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:56,332 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:56,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:56,432 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:56,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:56,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:56,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:56,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:56,435 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:56,435 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,435 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:56,435 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:56,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:56,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:56,443 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:56,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:56,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:56,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:56,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:56,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:56,445 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288896446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:56,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b32fdf9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:56,448 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:56,448 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:56,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:56,449 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:56,449 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:56,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288896447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288896448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,451 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,451 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,451 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:56,453 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:56,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:56,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:56,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:56,453 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:56,456 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,456 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,456 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,457 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,458 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,460 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,460 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:56,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:56,460 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,460 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,461 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:56,460 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,461 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:56,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=939 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/7e7b4fa117ed45a1b3d84d8b0ee998c6 2023-03-31 18:53:56,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=939 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/8a88885b51d34475965f43bb093ed5c2 2023-03-31 18:53:56,561 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:56,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:56,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:56,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:56,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:56,564 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:56,564 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,564 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:56,564 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:56,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,566 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,566 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,566 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,566 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:56,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:56,567 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:56,567 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,567 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:56,568 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,568 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:56,568 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:56,568 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,570 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:56,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eeb7ae1[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:56,571 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:56,572 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,574 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:56,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:56,574 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,574 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:56,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:56,578 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,578 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,578 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:56,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:56,581 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:56,581 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:56,581 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:56,581 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:56,581 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:56,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,583 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,584 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:56,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:56,586 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,586 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:56,586 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,687 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:56,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:56,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:56,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:56,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:56,690 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:56,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:56,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:56,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,691 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,693 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:56,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:56,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:56,694 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:56,694 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:56,695 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,695 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:56,695 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:56,695 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:56,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:56,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c0f1c6[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:56,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:56,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,700 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:56,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:56,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:56,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:56,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:56,707 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:56,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:56,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:56,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:56,707 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:56,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,712 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:56,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:56,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,714 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:56,714 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:56,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:56,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288896750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288896751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288896753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:56,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60008 deadline: 1680288896801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,808 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:53:56,814 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:56,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:56,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:56,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:56,818 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:56,818 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,818 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:56,818 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:56,818 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,818 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,821 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,821 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,821 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,821 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:56,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:56,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:56,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:56,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:56,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:56,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:56,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:56,823 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,823 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:56,823 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:56,823 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:56,825 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,825 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,825 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,825 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:56,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5de45d47[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:56,827 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:56,827 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:56,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:56,830 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:56,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:56,832 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:56,832 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:56,832 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:56,832 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:56,832 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:56,835 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,835 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:56,835 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:56,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,836 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,837 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:56,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:56,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:56,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:56,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:56,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,840 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:56,840 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:56,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:56,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:56,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=939 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/3f41cf23acf243eb82bc40672031b81f 2023-03-31 18:53:56,940 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:56,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:56,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/7e7b4fa117ed45a1b3d84d8b0ee998c6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/7e7b4fa117ed45a1b3d84d8b0ee998c6 2023-03-31 18:53:56,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:56,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:56,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:56,947 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:56,947 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,947 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:56,947 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:56,947 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,947 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,949 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:56,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:56,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/7e7b4fa117ed45a1b3d84d8b0ee998c6, entries=250, sequenceid=939, filesize=16.7 K 2023-03-31 18:53:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,950 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:56,950 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:56,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/8a88885b51d34475965f43bb093ed5c2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/8a88885b51d34475965f43bb093ed5c2 2023-03-31 18:53:56,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:56,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:56,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:56,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:56,951 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:56,952 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:56,952 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:56,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/8a88885b51d34475965f43bb093ed5c2, entries=150, sequenceid=939, filesize=11.9 K 2023-03-31 18:53:56,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/3f41cf23acf243eb82bc40672031b81f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/3f41cf23acf243eb82bc40672031b81f 2023-03-31 18:53:56,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/3f41cf23acf243eb82bc40672031b81f, entries=150, sequenceid=939, filesize=11.9 K 2023-03-31 18:53:56,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 087bf54664c6a240c659c71fb3782134 in 891ms, sequenceid=939, compaction requested=true 2023-03-31 18:53:56,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:56,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:56,968 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:56,968 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:56,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:56,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:56,970 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42188 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:56,970 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:56,970 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/A is initiating minor compaction (all files) 2023-03-31 18:53:56,970 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/B is initiating minor compaction (all files) 2023-03-31 18:53:56,970 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/A in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,970 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:56,970 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:56,970 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/B in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,971 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:56,971 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/98e62a18dbf7414b961ed3daf771373d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/86266f771f9d473abd2dd9d7068dd23d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/8a88885b51d34475965f43bb093ed5c2] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=36.4 K 2023-03-31 18:53:56,970 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/d8b8e3a4848648798d9c3eca8fdf3a6e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/00f07a1684754f1dae5cbdcd0ede3f98, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/7e7b4fa117ed45a1b3d84d8b0ee998c6] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=41.2 K 2023-03-31 18:53:56,971 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 98e62a18dbf7414b961ed3daf771373d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=890, earliestPutTs=1680288833781 2023-03-31 18:53:56,971 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:56,971 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-31 18:53:56,972 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 86266f771f9d473abd2dd9d7068dd23d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=925, earliestPutTs=1680288833962 2023-03-31 18:53:56,972 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d8b8e3a4848648798d9c3eca8fdf3a6e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=890, earliestPutTs=1680288833781 2023-03-31 18:53:56,972 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:56,973 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:56,973 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:56,973 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 00f07a1684754f1dae5cbdcd0ede3f98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=925, earliestPutTs=1680288833962 2023-03-31 18:53:56,973 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:56,973 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:56,973 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:56,973 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 8a88885b51d34475965f43bb093ed5c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=939, earliestPutTs=1680288836069 2023-03-31 18:53:56,973 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 7e7b4fa117ed45a1b3d84d8b0ee998c6, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=939, earliestPutTs=1680288836069 2023-03-31 18:53:57,004 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#B#compaction#74 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:57,011 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#A#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:57,058 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=976 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/22fae19c25d344e8a7749c1063cbfa78 2023-03-31 18:53:57,093 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=976 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/7a5f4f507fc543eeb8815ddcde2a184b 2023-03-31 18:53:57,146 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=976 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/0b1bd8f8374d417a8eedc1944892bfc3 2023-03-31 18:53:57,153 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/22fae19c25d344e8a7749c1063cbfa78 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/22fae19c25d344e8a7749c1063cbfa78 2023-03-31 18:53:57,160 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/22fae19c25d344e8a7749c1063cbfa78, entries=150, sequenceid=976, filesize=11.9 K 2023-03-31 18:53:57,167 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/7a5f4f507fc543eeb8815ddcde2a184b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/7a5f4f507fc543eeb8815ddcde2a184b 2023-03-31 18:53:57,174 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/7a5f4f507fc543eeb8815ddcde2a184b, entries=150, sequenceid=976, filesize=11.9 K 2023-03-31 18:53:57,175 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/0b1bd8f8374d417a8eedc1944892bfc3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/0b1bd8f8374d417a8eedc1944892bfc3 2023-03-31 18:53:57,180 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/0b1bd8f8374d417a8eedc1944892bfc3, entries=150, sequenceid=976, filesize=11.9 K 2023-03-31 18:53:57,181 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=0 B/0 for 087bf54664c6a240c659c71fb3782134 in 210ms, sequenceid=976, compaction requested=true 2023-03-31 18:53:57,181 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:57,181 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:57,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:57,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:57,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:57,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3155ae37[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:57,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:57,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,191 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,191 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:57,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:57,194 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:57,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:57,197 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:57,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:57,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:57,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:57,197 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:57,199 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,199 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,199 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,199 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,205 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,205 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,205 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,205 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:57,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:57,205 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,206 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:57,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,206 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:57,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:57,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:53:57,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:57,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:57,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:57,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:57,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:57,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:57,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=987 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/4aa2bafa58854d6da9ed07246dfdc1ec 2023-03-31 18:53:57,306 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:57,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:57,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:57,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:57,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:57,309 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:57,309 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,309 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:57,309 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:57,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680288837278, max timestamp=1680288837310Num uniques 859; 2023-03-31 18:53:57,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680288837278, max timestamp=1680288837310Num uniques 859; 2023-03-31 18:53:57,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680288837278, max timestamp=1680288837310Num uniques 859; 2023-03-31 18:53:57,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 18:53:57,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 18:53:57,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 18:53:57,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,312 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,313 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,313 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:57,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:57,315 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,315 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:57,315 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:57,315 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:57,317 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:57,319 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449cf68e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:57,319 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:57,321 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,321 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:57,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,324 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:57,324 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,324 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288897328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288897329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288897330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:57,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:57,336 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:57,336 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:57,336 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:57,336 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:57,336 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:57,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,344 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,346 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,346 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,346 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,346 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,347 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,346 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,347 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:57,347 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:57,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:57,347 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,347 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,347 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:57,348 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:57,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288897431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288897432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288897435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,448 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:57,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:57,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:57,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:57,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:57,452 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:57,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:57,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:57,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,455 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,455 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:57,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:57,462 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/bfbbdbd29dcc47969ad13d76af16d190 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/bfbbdbd29dcc47969ad13d76af16d190 2023-03-31 18:53:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:57,464 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,464 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:57,464 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:57,464 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:57,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,468 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,469 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,469 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,471 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/B of 087bf54664c6a240c659c71fb3782134 into bfbbdbd29dcc47969ad13d76af16d190(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:57,471 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:57,471 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/B, priority=13, startTime=1680288836968; duration=0sec 2023-03-31 18:53:57,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:57,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a6711a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:57,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:57,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,471 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/4e3d0bd2af704386b86ce1750ce429f5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4e3d0bd2af704386b86ce1750ce429f5 2023-03-31 18:53:57,472 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:57,472 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:53:57,474 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:57,475 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:53:57,475 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/C is initiating minor compaction (all files) 2023-03-31 18:53:57,475 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/C in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,475 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,475 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:57,476 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/c2fdef4056a044fbae8e55e26f9d42c5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/341488f66c914264a0b0c7db85ccddc9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/3f41cf23acf243eb82bc40672031b81f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/0b1bd8f8374d417a8eedc1944892bfc3] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=48.3 K 2023-03-31 18:53:57,476 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:57,476 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,477 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:57,477 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,477 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting c2fdef4056a044fbae8e55e26f9d42c5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=890, earliestPutTs=1680288833781 2023-03-31 18:53:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,478 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 341488f66c914264a0b0c7db85ccddc9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=925, earliestPutTs=1680288833962 2023-03-31 18:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,479 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3f41cf23acf243eb82bc40672031b81f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=939, earliestPutTs=1680288836069 2023-03-31 18:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,480 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 0b1bd8f8374d417a8eedc1944892bfc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=976, earliestPutTs=1680288836136 2023-03-31 18:53:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:57,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:57,481 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:57,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:57,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:57,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:57,481 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:57,482 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/A of 087bf54664c6a240c659c71fb3782134 into 4e3d0bd2af704386b86ce1750ce429f5(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:57,482 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:57,482 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/A, priority=13, startTime=1680288836968; duration=0sec 2023-03-31 18:53:57,482 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:57,483 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,483 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,483 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,484 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,486 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,489 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:57,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:57,489 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,489 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:57,490 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:57,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,494 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#C#compaction#81 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:57,590 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:57,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:57,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:57,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:57,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:57,594 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:57,594 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,594 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:57,594 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:57,594 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,594 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,596 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:57,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:57,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:57,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:57,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:57,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:57,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:57,598 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,598 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:57,598 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:57,599 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:57,600 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:57,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f571c2c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:57,603 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:57,603 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,605 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:57,605 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,605 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:57,606 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:57,606 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,606 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,607 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:57,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:57,610 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:57,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:57,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:57,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:57,610 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:57,611 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,611 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,611 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,613 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,614 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:57,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:57,621 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,622 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,622 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,621 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:57,622 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:57,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288897635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288897636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288897643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,690 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=987 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/cb4edf930934457886ad5f1d8c4d5496 2023-03-31 18:53:57,722 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:57,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:57,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:57,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:57,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:57,725 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:57,725 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,725 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:57,725 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:57,726 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,726 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,729 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,729 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,730 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,730 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:57,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:57,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:57,731 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,731 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:57,731 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:57,731 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:57,732 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:57,733 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,733 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,734 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,734 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:57,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@536f37e7[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:57,737 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:57,738 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,740 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,740 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:57,740 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,741 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:57,741 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:57,741 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:57,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:57,743 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:57,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:57,748 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:57,748 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:57,748 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:57,748 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:57,748 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:57,750 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,750 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,750 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,756 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,756 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:57,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:57,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,757 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:57,757 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:57,758 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,758 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,758 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,758 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,758 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,758 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,758 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,857 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:57,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:57,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:57,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:57,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:57,860 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:57,861 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,861 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:57,861 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:57,861 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,861 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,863 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,863 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,863 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:57,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:57,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:57,865 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,865 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:57,865 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:57,865 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:57,867 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70e48382[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:57,869 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:57,869 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,871 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:57,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:57,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:57,871 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,871 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:57,872 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,872 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,872 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,873 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:57,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:57,875 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:57,875 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:57,875 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:57,875 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:57,875 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:57,877 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,877 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,877 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,878 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,880 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,882 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,882 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:57,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:57,883 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,883 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,883 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,883 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,883 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:57,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:57,883 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,883 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:57,883 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,883 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:57,883 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:57,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:57,912 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/eff3e3b8a5c4454a9d4c0e3753b3f9d9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eff3e3b8a5c4454a9d4c0e3753b3f9d9 2023-03-31 18:53:57,920 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 087bf54664c6a240c659c71fb3782134/C of 087bf54664c6a240c659c71fb3782134 into eff3e3b8a5c4454a9d4c0e3753b3f9d9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:57,920 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:57,920 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/C, priority=12, startTime=1680288836969; duration=0sec 2023-03-31 18:53:57,920 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:57,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288897942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288897942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:57,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288897945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,983 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:57,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:57,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:57,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:57,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:57,987 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:57,987 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,987 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:57,987 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:57,987 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,987 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,988 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,988 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:57,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:57,989 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,989 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:57,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:57,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:57,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:57,990 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,990 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:57,990 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:57,990 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:57,992 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,992 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:57,996 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:57,996 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:57,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:57,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c90225[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:57,997 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:57,997 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,000 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,000 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,000 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:58,002 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:58,005 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:58,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:58,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:58,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:58,005 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:58,007 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,007 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,007 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:58,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:58,008 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,010 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,012 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,012 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:58,012 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:58,012 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,012 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,012 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,012 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:58,012 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:58,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:58,012 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:58,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:58,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:58,012 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:58,012 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:58,013 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:58,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:58,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:58,103 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=987 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/1ccc0e0bd7a0417fbd6acc7ff72c8a50 2023-03-31 18:53:58,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/4aa2bafa58854d6da9ed07246dfdc1ec as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4aa2bafa58854d6da9ed07246dfdc1ec 2023-03-31 18:53:58,113 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:58,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:58,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:58,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:58,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:58,116 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:58,116 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:58,116 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:58,116 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:58,116 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4aa2bafa58854d6da9ed07246dfdc1ec, entries=300, sequenceid=987, filesize=19.0 K 2023-03-31 18:53:58,117 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,117 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:58,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/cb4edf930934457886ad5f1d8c4d5496 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/cb4edf930934457886ad5f1d8c4d5496 2023-03-31 18:53:58,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:58,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:58,119 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,120 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:58,120 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:58,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:58,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:58,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:58,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:58,121 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:58,121 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:58,122 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:58,122 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:58,123 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/cb4edf930934457886ad5f1d8c4d5496, entries=150, sequenceid=987, filesize=11.9 K 2023-03-31 18:53:58,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,124 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:58,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:58,124 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:58,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/1ccc0e0bd7a0417fbd6acc7ff72c8a50 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1ccc0e0bd7a0417fbd6acc7ff72c8a50 2023-03-31 18:53:58,124 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:58,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:58,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408c60c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:58,127 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:58,127 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,132 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,132 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:58,132 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:58,132 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:58,132 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,132 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:58,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1ccc0e0bd7a0417fbd6acc7ff72c8a50, entries=150, sequenceid=987, filesize=11.9 K 2023-03-31 18:53:58,134 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:58,134 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 087bf54664c6a240c659c71fb3782134 in 870ms, sequenceid=987, compaction requested=true 2023-03-31 18:53:58,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:58,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:58,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:58,134 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:58,135 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:58,135 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:53:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:58,135 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:58,135 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:58,135 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,136 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44730 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,136 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/A is initiating minor compaction (all files) 2023-03-31 18:53:58,136 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:53:58,136 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/A in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:58,136 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/B is initiating minor compaction (all files) 2023-03-31 18:53:58,136 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4e3d0bd2af704386b86ce1750ce429f5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/22fae19c25d344e8a7749c1063cbfa78, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4aa2bafa58854d6da9ed07246dfdc1ec] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=43.7 K 2023-03-31 18:53:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:58,136 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/B in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:58,136 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/bfbbdbd29dcc47969ad13d76af16d190, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/7a5f4f507fc543eeb8815ddcde2a184b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/cb4edf930934457886ad5f1d8c4d5496] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=36.5 K 2023-03-31 18:53:58,137 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 4e3d0bd2af704386b86ce1750ce429f5, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=939, earliestPutTs=1680288836069 2023-03-31 18:53:58,137 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting bfbbdbd29dcc47969ad13d76af16d190, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=939, earliestPutTs=1680288836069 2023-03-31 18:53:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,137 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 22fae19c25d344e8a7749c1063cbfa78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=976, earliestPutTs=1680288836136 2023-03-31 18:53:58,137 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 7a5f4f507fc543eeb8815ddcde2a184b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=976, earliestPutTs=1680288836136 2023-03-31 18:53:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,138 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 4aa2bafa58854d6da9ed07246dfdc1ec, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1680288837258 2023-03-31 18:53:58,138 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting cb4edf930934457886ad5f1d8c4d5496, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1680288837258 2023-03-31 18:53:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:58,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:58,140 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:58,140 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:58,140 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:58,140 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:58,140 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:58,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,143 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:58,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:58,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:58,143 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,146 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:58,147 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#B#compaction#83 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:58,148 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#A#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:53:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:58,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:58,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:58,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:58,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:58,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:58,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:58,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,152 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:58,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:58,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:58,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:58,153 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:58,153 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:58,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:58,172 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/92de2e00947c4aa4a64e40701a778946 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/92de2e00947c4aa4a64e40701a778946 2023-03-31 18:53:58,181 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/A of 087bf54664c6a240c659c71fb3782134 into 92de2e00947c4aa4a64e40701a778946(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:58,181 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:58,181 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/A, priority=13, startTime=1680288838134; duration=0sec 2023-03-31 18:53:58,181 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:53:58,181 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 18:53:58,182 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 18:53:58,182 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 18:53:58,183 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. because compaction request was cancelled 2023-03-31 18:53:58,253 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:58,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:58,256 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:58,256 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:58,256 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:58,256 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:58,257 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,257 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:58,258 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:58,258 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:58,258 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,259 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:58,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:58,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:58,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:53:58,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:58,260 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:58,260 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:58,260 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-03-31 18:53:58,260 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:58,261 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:58,261 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:58,261 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:58,261 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:58,261 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:58,272 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1025 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/84e32737c3f44feeaf678f5e1da8be83 2023-03-31 18:53:58,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:58,450 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:58,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:58,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288898456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:58,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288898457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:58,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288898458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:58,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288898559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,561 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/deab86207eab4261830ba8e8ba9eca16 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/deab86207eab4261830ba8e8ba9eca16 2023-03-31 18:53:58,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:58,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288898561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:58,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288898563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,567 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 087bf54664c6a240c659c71fb3782134/B of 087bf54664c6a240c659c71fb3782134 into deab86207eab4261830ba8e8ba9eca16(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:53:58,567 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:58,567 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/B, priority=13, startTime=1680288838134; duration=0sec 2023-03-31 18:53:58,568 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:53:58,688 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1025 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/2002e803495b4693b177a31ec79593c1 2023-03-31 18:53:58,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:58,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288898763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:58,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288898766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:58,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:58,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288898767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288899066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288899069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288899071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,126 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1025 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/8ccb2814916d43c2bd17bce2f09bd2e5 2023-03-31 18:53:59,136 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/84e32737c3f44feeaf678f5e1da8be83 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84e32737c3f44feeaf678f5e1da8be83 2023-03-31 18:53:59,144 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84e32737c3f44feeaf678f5e1da8be83, entries=150, sequenceid=1025, filesize=11.9 K 2023-03-31 18:53:59,146 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/2002e803495b4693b177a31ec79593c1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2002e803495b4693b177a31ec79593c1 2023-03-31 18:53:59,158 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2002e803495b4693b177a31ec79593c1, entries=150, sequenceid=1025, filesize=11.9 K 2023-03-31 18:53:59,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/8ccb2814916d43c2bd17bce2f09bd2e5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8ccb2814916d43c2bd17bce2f09bd2e5 2023-03-31 18:53:59,168 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8ccb2814916d43c2bd17bce2f09bd2e5, entries=150, sequenceid=1025, filesize=11.9 K 2023-03-31 18:53:59,169 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 087bf54664c6a240c659c71fb3782134 in 909ms, sequenceid=1025, compaction requested=true 2023-03-31 18:53:59,169 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:59,170 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:59,173 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e721965[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:59,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:59,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:59,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:53:59,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:59,184 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:59,184 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:59,184 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:59,184 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:59,184 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:59,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:59,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:59,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:59,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:59,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:59,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:59,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:59,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:59,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:59,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:59,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:59,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,196 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:59,197 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:59,196 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:59,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:59,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:59,315 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:53:59,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:53:59,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:53:59,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:53:59,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:53:59,332 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:53:59,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:59,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:53:59,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:53:59,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:59,336 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:59,336 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:59,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:53:59,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:53:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:59,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:53:59,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:53:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:53:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:53:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:53:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:53:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:53:59,355 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:59,356 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:53:59,356 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:53:59,361 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:59,362 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:59,362 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:59,362 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:59,362 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:59,362 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:53:59,398 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1037 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/9a92b5842578465f8bbd36429fa49de3 2023-03-31 18:53:59,421 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1037 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/f2f8c3a4269243659741ae592af602aa 2023-03-31 18:53:59,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:59,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:53:59,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288899592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288839577, max timestamp=1680288839594Num uniques 859; 2023-03-31 18:53:59,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:53:59,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288839577, max timestamp=1680288839594Num uniques 859; 2023-03-31 18:53:59,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:53:59,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288839577, max timestamp=1680288839594Num uniques 859; 2023-03-31 18:53:59,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:53:59,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288899621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288899623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288899697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288899724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288899725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,881 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1037 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/d911784f0f134f52ab409102038851c0 2023-03-31 18:53:59,889 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/9a92b5842578465f8bbd36429fa49de3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9a92b5842578465f8bbd36429fa49de3 2023-03-31 18:53:59,895 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9a92b5842578465f8bbd36429fa49de3, entries=100, sequenceid=1037, filesize=9.5 K 2023-03-31 18:53:59,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/f2f8c3a4269243659741ae592af602aa as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f2f8c3a4269243659741ae592af602aa 2023-03-31 18:53:59,901 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f2f8c3a4269243659741ae592af602aa, entries=100, sequenceid=1037, filesize=9.5 K 2023-03-31 18:53:59,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/d911784f0f134f52ab409102038851c0 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d911784f0f134f52ab409102038851c0 2023-03-31 18:53:59,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288899902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,913 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d911784f0f134f52ab409102038851c0, entries=100, sequenceid=1037, filesize=9.5 K 2023-03-31 18:53:59,913 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 087bf54664c6a240c659c71fb3782134 in 557ms, sequenceid=1037, compaction requested=true 2023-03-31 18:53:59,914 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:53:59,914 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:53:59,914 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:53:59,914 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:53:59,914 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,914 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:53:59,914 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:53:59,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:59,916 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:59,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ade67ba[Count = 0] remaining members to acquire global barrier 2023-03-31 18:53:59,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:53:59,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,919 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:53:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:53:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:53:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:53:59,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:59,921 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:53:59,921 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:53:59,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:53:59,922 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:53:59,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:53:59,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:53:59,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:53:59,922 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:53:59,923 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,923 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:53:59,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,923 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:59,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:59,924 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:53:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:59,926 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:53:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:53:59,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:59,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:53:59,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:53:59,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:53:59,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:53:59,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:53:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:53:59,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:53:59,929 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:53:59,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:53:59,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:53:59,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:53:59,929 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:53:59,929 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:53:59,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:53:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:53:59,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:53:59,933 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-31 18:53:59,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:53:59,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:59,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:53:59,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:59,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:53:59,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:53:59,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288899948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:53:59,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:53:59,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288899948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,029 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:00,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:00,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:00,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:00,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:00,032 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:00,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:00,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:00,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:00,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,034 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,034 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:00,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:00,034 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,034 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:00,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:00,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:00,036 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:00,036 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:54:00,036 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:54:00,036 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,040 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413d8aee[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:00,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:00,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,043 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:00,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:00,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:00,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:00,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:00,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:00,048 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:00,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:00,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:00,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:00,048 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:00,049 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,049 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,049 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,050 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:00,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288900049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:00,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288900050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:00,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:00,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:00,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:00,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:00,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:00,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:00,056 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:00,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,056 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:00,056 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:00,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:00,156 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:00,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:00,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:00,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:00,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:00,160 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:00,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:00,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:00,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:00,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:00,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:00,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:00,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:00,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:00,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:00,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:00,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:00,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:00,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:00,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:00,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:54:00,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:54:00,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:00,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:00,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:00,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:00,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:00,166 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:00,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3470c63[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:00,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,169 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:00,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:00,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:00,169 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,169 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:00,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:00,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:00,174 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:00,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:00,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:00,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:00,174 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:00,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:00,176 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:00,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:00,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:00,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:00,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:00,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:00,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:00,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:00,180 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:00,180 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:00,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:00,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288900206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:00,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288900252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:00,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288900253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,280 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:00,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:00,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:00,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:00,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:00,283 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:00,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:00,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:00,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:00,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:00,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:00,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:00,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:00,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:00,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:00,287 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:00,287 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:54:00,287 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:54:00,287 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,292 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dc0945f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:00,294 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:00,294 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,295 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:00,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:00,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:00,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:00,297 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:00,297 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:00,297 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:00,298 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:00,300 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:00,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:00,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:00,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:00,300 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:00,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:00,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,304 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:00,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:00,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:00,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,309 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:00,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:00,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:00,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:00,309 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:00,309 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:00,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:00,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1073 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/a69b0c5e65dc4596b7a0aa811447041d 2023-03-31 18:54:00,409 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:00,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:00,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:00,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:00,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:00,413 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:00,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:00,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:00,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:00,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,414 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:00,414 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:00,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:00,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:00,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:00,416 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:00,416 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:54:00,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:54:00,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:00,418 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,418 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,419 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,419 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:00,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@342ad7ea[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:00,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,428 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:00,429 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,429 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:00,433 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:00,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:00,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:00,438 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:00,438 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:00,438 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:00,438 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:00,438 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:00,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:00,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:00,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:00,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,443 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:00,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:00,450 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:00,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:00,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:00,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:00,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:00,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,450 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:00,450 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:00,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:00,550 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:00,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:00,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:00,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:00,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:00,554 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:00,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:00,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:00,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:00,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:00,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:00,556 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,556 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,556 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:00,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:00,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:00,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288900555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:00,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:00,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:00,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:00,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:00,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:00,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. started... 2023-03-31 18:54:00,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. as already flushing 2023-03-31 18:54:00,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:00,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:00,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:00,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:00,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:00,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:00,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288900556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:00,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@196d517b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:00,562 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:00,562 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,563 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:00,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:00,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:00,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:00,573 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:00,573 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:00,573 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:00,573 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:00,573 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:00,574 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,574 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:00,574 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:00,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:00,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:00,575 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,576 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:00,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:00,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:00,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:00,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:00,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:00,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:00,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:00,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:00,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:00,580 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:00,580 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:00,640 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:63858 2023-03-31 18:54:00,640 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:00,643 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4808c6d5 to 127.0.0.1:63858 2023-03-31 18:54:00,643 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:63858 2023-03-31 18:54:00,643 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:00,643 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:00,649 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b02eeb3 to 127.0.0.1:63858 2023-03-31 18:54:00,649 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:00,680 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:00,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:00,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:00,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59988 deadline: 1680288900711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1073 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/5e7d0b51cb9e44aaa83075909cb7ec09 2023-03-31 18:54:00,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:00,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60008 deadline: 1680288900833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:00,835 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8185 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:01,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:01,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60006 deadline: 1680288901060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:01,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:01,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 423 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60050 deadline: 1680288901063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:01,199 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1073 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/1dd86145b5e34626a98af69a860f3ec6 2023-03-31 18:54:01,204 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/a69b0c5e65dc4596b7a0aa811447041d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a69b0c5e65dc4596b7a0aa811447041d 2023-03-31 18:54:01,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a69b0c5e65dc4596b7a0aa811447041d, entries=200, sequenceid=1073, filesize=14.3 K 2023-03-31 18:54:01,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/5e7d0b51cb9e44aaa83075909cb7ec09 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/5e7d0b51cb9e44aaa83075909cb7ec09 2023-03-31 18:54:01,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/5e7d0b51cb9e44aaa83075909cb7ec09, entries=150, sequenceid=1073, filesize=11.9 K 2023-03-31 18:54:01,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/1dd86145b5e34626a98af69a860f3ec6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1dd86145b5e34626a98af69a860f3ec6 2023-03-31 18:54:01,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1dd86145b5e34626a98af69a860f3ec6, entries=150, sequenceid=1073, filesize=11.9 K 2023-03-31 18:54:01,219 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 087bf54664c6a240c659c71fb3782134 in 1286ms, sequenceid=1073, compaction requested=true 2023-03-31 18:54:01,219 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:54:01,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:01,219 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:54:01,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:01,219 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:54:01,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:01,220 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:54:01,221 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/A is initiating minor compaction (all files) 2023-03-31 18:54:01,221 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47254 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:54:01,221 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/A in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:01,221 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/B is initiating minor compaction (all files) 2023-03-31 18:54:01,221 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/92de2e00947c4aa4a64e40701a778946, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84e32737c3f44feeaf678f5e1da8be83, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9a92b5842578465f8bbd36429fa49de3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a69b0c5e65dc4596b7a0aa811447041d] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=48.5 K 2023-03-31 18:54:01,221 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/B in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:01,221 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/deab86207eab4261830ba8e8ba9eca16, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2002e803495b4693b177a31ec79593c1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f2f8c3a4269243659741ae592af602aa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/5e7d0b51cb9e44aaa83075909cb7ec09] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=46.1 K 2023-03-31 18:54:01,221 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 92de2e00947c4aa4a64e40701a778946, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1680288837258 2023-03-31 18:54:01,221 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting deab86207eab4261830ba8e8ba9eca16, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1680288837258 2023-03-31 18:54:01,221 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 84e32737c3f44feeaf678f5e1da8be83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1025, earliestPutTs=1680288837318 2023-03-31 18:54:01,222 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2002e803495b4693b177a31ec79593c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1025, earliestPutTs=1680288837318 2023-03-31 18:54:01,222 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9a92b5842578465f8bbd36429fa49de3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1037, earliestPutTs=1680288838454 2023-03-31 18:54:01,222 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f2f8c3a4269243659741ae592af602aa, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1037, earliestPutTs=1680288838454 2023-03-31 18:54:01,222 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a69b0c5e65dc4596b7a0aa811447041d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1073, earliestPutTs=1680288839619 2023-03-31 18:54:01,222 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 5e7d0b51cb9e44aaa83075909cb7ec09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1073, earliestPutTs=1680288839619 2023-03-31 18:54:01,231 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#B#compaction#95 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:01,232 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#A#compaction#94 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:01,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 087bf54664c6a240c659c71fb3782134 2023-03-31 18:54:01,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:54:01,577 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:63858 2023-03-31 18:54:01,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:54:01,577 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:01,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:01,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:54:01,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:01,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:54:01,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:01,650 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/f11fe0b5dab045dc9d589919a6353a7e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/f11fe0b5dab045dc9d589919a6353a7e 2023-03-31 18:54:01,650 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/c47a2c5021e64022bb29633c58ffc9fe as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/c47a2c5021e64022bb29633c58ffc9fe 2023-03-31 18:54:01,656 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 087bf54664c6a240c659c71fb3782134/A of 087bf54664c6a240c659c71fb3782134 into f11fe0b5dab045dc9d589919a6353a7e(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:01,656 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:54:01,656 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/A, priority=12, startTime=1680288841219; duration=0sec 2023-03-31 18:54:01,656 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:01,656 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:54:01,656 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 087bf54664c6a240c659c71fb3782134/B of 087bf54664c6a240c659c71fb3782134 into c47a2c5021e64022bb29633c58ffc9fe(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:01,656 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:54:01,656 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/B, priority=12, startTime=1680288841219; duration=0sec 2023-03-31 18:54:01,656 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:01,658 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59338 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:54:01,658 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 087bf54664c6a240c659c71fb3782134/C is initiating minor compaction (all files) 2023-03-31 18:54:01,658 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 087bf54664c6a240c659c71fb3782134/C in TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:01,658 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eff3e3b8a5c4454a9d4c0e3753b3f9d9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1ccc0e0bd7a0417fbd6acc7ff72c8a50, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8ccb2814916d43c2bd17bce2f09bd2e5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d911784f0f134f52ab409102038851c0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1dd86145b5e34626a98af69a860f3ec6] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp, totalSize=57.9 K 2023-03-31 18:54:01,659 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting eff3e3b8a5c4454a9d4c0e3753b3f9d9, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=976, earliestPutTs=1680288836136 2023-03-31 18:54:01,659 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 1ccc0e0bd7a0417fbd6acc7ff72c8a50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1680288837258 2023-03-31 18:54:01,659 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8ccb2814916d43c2bd17bce2f09bd2e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1025, earliestPutTs=1680288837318 2023-03-31 18:54:01,660 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d911784f0f134f52ab409102038851c0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1037, earliestPutTs=1680288838454 2023-03-31 18:54:01,660 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 1dd86145b5e34626a98af69a860f3ec6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1073, earliestPutTs=1680288839619 2023-03-31 18:54:01,670 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 087bf54664c6a240c659c71fb3782134#C#compaction#97 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:01,713 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63859c62 to 127.0.0.1:63858 2023-03-31 18:54:01,713 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:01,985 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1083 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/c99f36df51a34b98929ea19894c3b6ba 2023-03-31 18:54:02,069 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:63858 2023-03-31 18:54:02,069 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:02,070 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:63858 2023-03-31 18:54:02,070 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:02,080 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/4b1718dcd67041af9f7e359301f03fe1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/4b1718dcd67041af9f7e359301f03fe1 2023-03-31 18:54:02,085 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 087bf54664c6a240c659c71fb3782134/C of 087bf54664c6a240c659c71fb3782134 into 4b1718dcd67041af9f7e359301f03fe1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:02,085 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:54:02,085 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134., storeName=087bf54664c6a240c659c71fb3782134/C, priority=11, startTime=1680288841219; duration=0sec 2023-03-31 18:54:02,085 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:02,397 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1083 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/5bf5f5cd9ecf46c4b03302a2ad07aa83 2023-03-31 18:54:02,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1083 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/5cc793d25b9b492d9795cd3ec062b201 2023-03-31 18:54:02,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/c99f36df51a34b98929ea19894c3b6ba as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/c99f36df51a34b98929ea19894c3b6ba 2023-03-31 18:54:02,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/c99f36df51a34b98929ea19894c3b6ba, entries=150, sequenceid=1083, filesize=11.9 K 2023-03-31 18:54:02,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/5bf5f5cd9ecf46c4b03302a2ad07aa83 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/5bf5f5cd9ecf46c4b03302a2ad07aa83 2023-03-31 18:54:02,823 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/5bf5f5cd9ecf46c4b03302a2ad07aa83, entries=150, sequenceid=1083, filesize=11.9 K 2023-03-31 18:54:02,824 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/5cc793d25b9b492d9795cd3ec062b201 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/5cc793d25b9b492d9795cd3ec062b201 2023-03-31 18:54:02,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/5cc793d25b9b492d9795cd3ec062b201, entries=150, sequenceid=1083, filesize=11.9 K 2023-03-31 18:54:02,829 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=20.13 KB/20610 for 087bf54664c6a240c659c71fb3782134 in 1252ms, sequenceid=1083, compaction requested=false 2023-03-31 18:54:02,829 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:54:10,936 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42927198 to 127.0.0.1:63858 2023-03-31 18:54:10,936 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:10,936 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 18:54:10,936 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 293 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 183 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 93 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 313 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 105 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 4472 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 4519 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1975 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5921 rows 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1959 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5872 rows 2023-03-31 18:54:10,937 INFO [Listener at localhost/43327] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 18:54:10,937 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35594b1b to 127.0.0.1:63858 2023-03-31 18:54:10,938 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:10,943 INFO [Listener at localhost/43327] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 18:54:10,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 18:54:10,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:10,957 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288850957"}]},"ts":"1680288850957"} 2023-03-31 18:54:10,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 18:54:10,958 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 18:54:10,960 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 18:54:10,961 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=087bf54664c6a240c659c71fb3782134, UNASSIGN}] 2023-03-31 18:54:10,962 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=087bf54664c6a240c659c71fb3782134, UNASSIGN 2023-03-31 18:54:10,962 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=087bf54664c6a240c659c71fb3782134, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:10,963 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288850962"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288850962"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288850962"}]},"ts":"1680288850962"} 2023-03-31 18:54:10,964 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:54:11,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 18:54:11,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 087bf54664c6a240c659c71fb3782134 2023-03-31 18:54:11,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 087bf54664c6a240c659c71fb3782134, disabling compactions & flushes 2023-03-31 18:54:11,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:11,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:11,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. after waiting 0 ms 2023-03-31 18:54:11,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:11,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 087bf54664c6a240c659c71fb3782134 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 18:54:11,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=A 2023-03-31 18:54:11,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:11,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=B 2023-03-31 18:54:11,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:11,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 087bf54664c6a240c659c71fb3782134, store=C 2023-03-31 18:54:11,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:11,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 18:54:11,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1093 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/186d423d272d407c80a6c155778cec92 2023-03-31 18:54:11,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 18:54:11,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1093 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/d41905b41465463d8427605da8af3eb0 2023-03-31 18:54:12,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 18:54:12,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1093 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/88a4015466f241339e58d698701194bd 2023-03-31 18:54:12,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/A/186d423d272d407c80a6c155778cec92 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/186d423d272d407c80a6c155778cec92 2023-03-31 18:54:12,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/186d423d272d407c80a6c155778cec92, entries=150, sequenceid=1093, filesize=11.9 K 2023-03-31 18:54:12,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/B/d41905b41465463d8427605da8af3eb0 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/d41905b41465463d8427605da8af3eb0 2023-03-31 18:54:12,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/d41905b41465463d8427605da8af3eb0, entries=150, sequenceid=1093, filesize=11.9 K 2023-03-31 18:54:12,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/.tmp/C/88a4015466f241339e58d698701194bd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/88a4015466f241339e58d698701194bd 2023-03-31 18:54:12,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/88a4015466f241339e58d698701194bd, entries=150, sequenceid=1093, filesize=11.9 K 2023-03-31 18:54:12,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 087bf54664c6a240c659c71fb3782134 in 1252ms, sequenceid=1093, compaction requested=true 2023-03-31 18:54:12,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/99d14cfdd76a40008da678a335636065, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/dca0460573744b9c97a83f8f2818d6b5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/217d5f494fb041559ccde2d1895f273d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/6b4f4bf25ea74dd09f4eb6baa8f6ea6f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a33f517bb759403798a5835293835d2a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/c33bc7398d7e462289b598635265c7f1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/727adf9ff5934ea39f3797f3dbcaad33, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/58c16491d6a04565b79fe0f88473dfe3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/695915fe34d14e7da4a77f36c8fe6e06, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2e36e654f6af4d87a51b6aa238274ba4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/3c78df96cefb47758fdf5e4379aa1d69, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/8a122b3bc14e4649bfbd44cb50590cdc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/87ca19054e6b4be1ac516a1637aac3f4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/bb532030c24d41f1b0300a46ffe8ddf6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/e7b890bf4a7744b9a912ed420a8f6cb2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/cd3d4c3d58cc485e935489f002efad72, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/11e1297e94d745ef8afefd26cb5826e8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9f866d746ebe49609a6429b27972c337, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/70dff6fb8d974ff8be84946e0ce5d6d4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/83c6304bfd574d2e921a68b7db8b9769, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2d443de1f30f43d59049682c0008d46c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/d8b8e3a4848648798d9c3eca8fdf3a6e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84ee81901db442e690dc23f87c50375e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/00f07a1684754f1dae5cbdcd0ede3f98, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/7e7b4fa117ed45a1b3d84d8b0ee998c6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4e3d0bd2af704386b86ce1750ce429f5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/22fae19c25d344e8a7749c1063cbfa78, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4aa2bafa58854d6da9ed07246dfdc1ec, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/92de2e00947c4aa4a64e40701a778946, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84e32737c3f44feeaf678f5e1da8be83, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9a92b5842578465f8bbd36429fa49de3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a69b0c5e65dc4596b7a0aa811447041d] to archive 2023-03-31 18:54:12,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:54:12,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/99d14cfdd76a40008da678a335636065 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/99d14cfdd76a40008da678a335636065 2023-03-31 18:54:12,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/dca0460573744b9c97a83f8f2818d6b5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/dca0460573744b9c97a83f8f2818d6b5 2023-03-31 18:54:12,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/217d5f494fb041559ccde2d1895f273d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/217d5f494fb041559ccde2d1895f273d 2023-03-31 18:54:12,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/6b4f4bf25ea74dd09f4eb6baa8f6ea6f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/6b4f4bf25ea74dd09f4eb6baa8f6ea6f 2023-03-31 18:54:12,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a33f517bb759403798a5835293835d2a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a33f517bb759403798a5835293835d2a 2023-03-31 18:54:12,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/c33bc7398d7e462289b598635265c7f1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/c33bc7398d7e462289b598635265c7f1 2023-03-31 18:54:12,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/727adf9ff5934ea39f3797f3dbcaad33 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/727adf9ff5934ea39f3797f3dbcaad33 2023-03-31 18:54:12,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/58c16491d6a04565b79fe0f88473dfe3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/58c16491d6a04565b79fe0f88473dfe3 2023-03-31 18:54:12,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/695915fe34d14e7da4a77f36c8fe6e06 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/695915fe34d14e7da4a77f36c8fe6e06 2023-03-31 18:54:12,414 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2e36e654f6af4d87a51b6aa238274ba4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2e36e654f6af4d87a51b6aa238274ba4 2023-03-31 18:54:12,415 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/3c78df96cefb47758fdf5e4379aa1d69 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/3c78df96cefb47758fdf5e4379aa1d69 2023-03-31 18:54:12,417 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/8a122b3bc14e4649bfbd44cb50590cdc to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/8a122b3bc14e4649bfbd44cb50590cdc 2023-03-31 18:54:12,418 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/87ca19054e6b4be1ac516a1637aac3f4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/87ca19054e6b4be1ac516a1637aac3f4 2023-03-31 18:54:12,419 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/bb532030c24d41f1b0300a46ffe8ddf6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/bb532030c24d41f1b0300a46ffe8ddf6 2023-03-31 18:54:12,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/e7b890bf4a7744b9a912ed420a8f6cb2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/e7b890bf4a7744b9a912ed420a8f6cb2 2023-03-31 18:54:12,422 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/cd3d4c3d58cc485e935489f002efad72 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/cd3d4c3d58cc485e935489f002efad72 2023-03-31 18:54:12,423 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/11e1297e94d745ef8afefd26cb5826e8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/11e1297e94d745ef8afefd26cb5826e8 2023-03-31 18:54:12,424 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9f866d746ebe49609a6429b27972c337 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9f866d746ebe49609a6429b27972c337 2023-03-31 18:54:12,426 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/70dff6fb8d974ff8be84946e0ce5d6d4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/70dff6fb8d974ff8be84946e0ce5d6d4 2023-03-31 18:54:12,427 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/83c6304bfd574d2e921a68b7db8b9769 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/83c6304bfd574d2e921a68b7db8b9769 2023-03-31 18:54:12,428 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2d443de1f30f43d59049682c0008d46c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/2d443de1f30f43d59049682c0008d46c 2023-03-31 18:54:12,429 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/d8b8e3a4848648798d9c3eca8fdf3a6e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/d8b8e3a4848648798d9c3eca8fdf3a6e 2023-03-31 18:54:12,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84ee81901db442e690dc23f87c50375e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84ee81901db442e690dc23f87c50375e 2023-03-31 18:54:12,431 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/00f07a1684754f1dae5cbdcd0ede3f98 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/00f07a1684754f1dae5cbdcd0ede3f98 2023-03-31 18:54:12,433 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/7e7b4fa117ed45a1b3d84d8b0ee998c6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/7e7b4fa117ed45a1b3d84d8b0ee998c6 2023-03-31 18:54:12,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4e3d0bd2af704386b86ce1750ce429f5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4e3d0bd2af704386b86ce1750ce429f5 2023-03-31 18:54:12,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/22fae19c25d344e8a7749c1063cbfa78 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/22fae19c25d344e8a7749c1063cbfa78 2023-03-31 18:54:12,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4aa2bafa58854d6da9ed07246dfdc1ec to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/4aa2bafa58854d6da9ed07246dfdc1ec 2023-03-31 18:54:12,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/92de2e00947c4aa4a64e40701a778946 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/92de2e00947c4aa4a64e40701a778946 2023-03-31 18:54:12,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84e32737c3f44feeaf678f5e1da8be83 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/84e32737c3f44feeaf678f5e1da8be83 2023-03-31 18:54:12,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9a92b5842578465f8bbd36429fa49de3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/9a92b5842578465f8bbd36429fa49de3 2023-03-31 18:54:12,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a69b0c5e65dc4596b7a0aa811447041d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/a69b0c5e65dc4596b7a0aa811447041d 2023-03-31 18:54:12,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/6aef52c4376d46d2a48b7e2334e1611b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b0d905e4bb294adc900219a0ec825dbd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/d975dc679ba041f99c37d7daa8b41364, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/af6bd5c42ed4426b96c69327d6355be3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b8e649b19a124969918d9c9174005cb4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/73ec365395de46d589c250c183bce43d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3bffb0c94c0d44788409ed6e75f85c0c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/945b63273535445aa65ce78146c534f7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/1d3f12ba99324e539621a40982481351, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2255640946054dd1bc8d1b4237795e6f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/23562af769be4e158c6b50cd797b3255, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f84e4521b2244e8f9b6ffb68a6a5d53f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f3d009ec29c54c9f906c9e2bf4c79346, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/4a20eb53f94b44fe82768e40185e86c6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/26a6dc476cb84f8d99f13d41ab8464ac, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3c470a11e17842e9b1d3d7c81f0717a0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/45c84078f8bd47f4929b7f4c949e112c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f788252a649d4789b060bc17f8147fa9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/af3031bbcd2540be84bf0700fd546cb3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/ded953bb6cf240ffbe58425b9676ae09, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b7a22742978c4f6d927db02b69b5c22d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/98e62a18dbf7414b961ed3daf771373d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/19aac34c89fc479d8e0dda0d7f71173d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/86266f771f9d473abd2dd9d7068dd23d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/bfbbdbd29dcc47969ad13d76af16d190, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/8a88885b51d34475965f43bb093ed5c2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/7a5f4f507fc543eeb8815ddcde2a184b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/deab86207eab4261830ba8e8ba9eca16, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/cb4edf930934457886ad5f1d8c4d5496, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2002e803495b4693b177a31ec79593c1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f2f8c3a4269243659741ae592af602aa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/5e7d0b51cb9e44aaa83075909cb7ec09] to archive 2023-03-31 18:54:12,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:54:12,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/6aef52c4376d46d2a48b7e2334e1611b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/6aef52c4376d46d2a48b7e2334e1611b 2023-03-31 18:54:12,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b0d905e4bb294adc900219a0ec825dbd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b0d905e4bb294adc900219a0ec825dbd 2023-03-31 18:54:12,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/d975dc679ba041f99c37d7daa8b41364 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/d975dc679ba041f99c37d7daa8b41364 2023-03-31 18:54:12,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/af6bd5c42ed4426b96c69327d6355be3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/af6bd5c42ed4426b96c69327d6355be3 2023-03-31 18:54:12,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b8e649b19a124969918d9c9174005cb4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b8e649b19a124969918d9c9174005cb4 2023-03-31 18:54:12,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/73ec365395de46d589c250c183bce43d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/73ec365395de46d589c250c183bce43d 2023-03-31 18:54:12,473 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3bffb0c94c0d44788409ed6e75f85c0c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3bffb0c94c0d44788409ed6e75f85c0c 2023-03-31 18:54:12,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/945b63273535445aa65ce78146c534f7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/945b63273535445aa65ce78146c534f7 2023-03-31 18:54:12,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/1d3f12ba99324e539621a40982481351 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/1d3f12ba99324e539621a40982481351 2023-03-31 18:54:12,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2255640946054dd1bc8d1b4237795e6f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2255640946054dd1bc8d1b4237795e6f 2023-03-31 18:54:12,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/23562af769be4e158c6b50cd797b3255 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/23562af769be4e158c6b50cd797b3255 2023-03-31 18:54:12,478 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f84e4521b2244e8f9b6ffb68a6a5d53f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f84e4521b2244e8f9b6ffb68a6a5d53f 2023-03-31 18:54:12,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f3d009ec29c54c9f906c9e2bf4c79346 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f3d009ec29c54c9f906c9e2bf4c79346 2023-03-31 18:54:12,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/4a20eb53f94b44fe82768e40185e86c6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/4a20eb53f94b44fe82768e40185e86c6 2023-03-31 18:54:12,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/26a6dc476cb84f8d99f13d41ab8464ac to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/26a6dc476cb84f8d99f13d41ab8464ac 2023-03-31 18:54:12,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3c470a11e17842e9b1d3d7c81f0717a0 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/3c470a11e17842e9b1d3d7c81f0717a0 2023-03-31 18:54:12,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/45c84078f8bd47f4929b7f4c949e112c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/45c84078f8bd47f4929b7f4c949e112c 2023-03-31 18:54:12,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f788252a649d4789b060bc17f8147fa9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f788252a649d4789b060bc17f8147fa9 2023-03-31 18:54:12,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/af3031bbcd2540be84bf0700fd546cb3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/af3031bbcd2540be84bf0700fd546cb3 2023-03-31 18:54:12,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/ded953bb6cf240ffbe58425b9676ae09 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/ded953bb6cf240ffbe58425b9676ae09 2023-03-31 18:54:12,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b7a22742978c4f6d927db02b69b5c22d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/b7a22742978c4f6d927db02b69b5c22d 2023-03-31 18:54:12,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/98e62a18dbf7414b961ed3daf771373d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/98e62a18dbf7414b961ed3daf771373d 2023-03-31 18:54:12,490 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/19aac34c89fc479d8e0dda0d7f71173d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/19aac34c89fc479d8e0dda0d7f71173d 2023-03-31 18:54:12,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/86266f771f9d473abd2dd9d7068dd23d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/86266f771f9d473abd2dd9d7068dd23d 2023-03-31 18:54:12,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/bfbbdbd29dcc47969ad13d76af16d190 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/bfbbdbd29dcc47969ad13d76af16d190 2023-03-31 18:54:12,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/8a88885b51d34475965f43bb093ed5c2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/8a88885b51d34475965f43bb093ed5c2 2023-03-31 18:54:12,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/7a5f4f507fc543eeb8815ddcde2a184b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/7a5f4f507fc543eeb8815ddcde2a184b 2023-03-31 18:54:12,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/deab86207eab4261830ba8e8ba9eca16 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/deab86207eab4261830ba8e8ba9eca16 2023-03-31 18:54:12,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/cb4edf930934457886ad5f1d8c4d5496 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/cb4edf930934457886ad5f1d8c4d5496 2023-03-31 18:54:12,498 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2002e803495b4693b177a31ec79593c1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/2002e803495b4693b177a31ec79593c1 2023-03-31 18:54:12,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f2f8c3a4269243659741ae592af602aa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/f2f8c3a4269243659741ae592af602aa 2023-03-31 18:54:12,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/5e7d0b51cb9e44aaa83075909cb7ec09 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/5e7d0b51cb9e44aaa83075909cb7ec09 2023-03-31 18:54:12,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/e8169e04ac1e40cab631c3d41f2f083d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/52468637d50d456faac82725e2afaf86, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/409c65b545e946079f68ad1cac833a55, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/88a6c6d7283742b0a7164cb42c8099c2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/348d0e867f9345d5942a2c32b9cd4ae9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/66d85dbb216c431b96806e4287d38c5e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/41ecd0957d824bb79b2f7501342a899f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/4d1602af55a244afb4d0966ab455d036, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8a94c494b0b4470bafaf6bb797e2b237, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/f7815818dfe8490b89e8650d70ee8fae, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ec885308c4f5487b93003a43d84d56cd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/9967b5ae49fa4ebaad3882af54dc7225, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/b2645fcfed0a43b3a25ee047389a2736, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/780fa56fc9fe4b4ead088ea0e2bd5359, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/086d5c4031994d42b207f6118b6a0a55, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eb289c72dc354401accc57c8bcd73260, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ee3fd02fbd324d63a599b7301fa0c157, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/db4f92f213a34a279b4ef7db155ef910, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d9179821e57f448c95eac24794299919, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/7a02aad6ac5d49f3b389d23c75e51cd7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/c2fdef4056a044fbae8e55e26f9d42c5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/2b1470cdf7774b9d84325b07f5081ee0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/341488f66c914264a0b0c7db85ccddc9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/3f41cf23acf243eb82bc40672031b81f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eff3e3b8a5c4454a9d4c0e3753b3f9d9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/0b1bd8f8374d417a8eedc1944892bfc3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1ccc0e0bd7a0417fbd6acc7ff72c8a50, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8ccb2814916d43c2bd17bce2f09bd2e5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d911784f0f134f52ab409102038851c0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1dd86145b5e34626a98af69a860f3ec6] to archive 2023-03-31 18:54:12,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:54:12,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/e8169e04ac1e40cab631c3d41f2f083d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/e8169e04ac1e40cab631c3d41f2f083d 2023-03-31 18:54:12,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/52468637d50d456faac82725e2afaf86 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/52468637d50d456faac82725e2afaf86 2023-03-31 18:54:12,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/409c65b545e946079f68ad1cac833a55 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/409c65b545e946079f68ad1cac833a55 2023-03-31 18:54:12,510 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/88a6c6d7283742b0a7164cb42c8099c2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/88a6c6d7283742b0a7164cb42c8099c2 2023-03-31 18:54:12,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/348d0e867f9345d5942a2c32b9cd4ae9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/348d0e867f9345d5942a2c32b9cd4ae9 2023-03-31 18:54:12,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/66d85dbb216c431b96806e4287d38c5e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/66d85dbb216c431b96806e4287d38c5e 2023-03-31 18:54:12,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/41ecd0957d824bb79b2f7501342a899f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/41ecd0957d824bb79b2f7501342a899f 2023-03-31 18:54:12,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/4d1602af55a244afb4d0966ab455d036 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/4d1602af55a244afb4d0966ab455d036 2023-03-31 18:54:12,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8a94c494b0b4470bafaf6bb797e2b237 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8a94c494b0b4470bafaf6bb797e2b237 2023-03-31 18:54:12,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/f7815818dfe8490b89e8650d70ee8fae to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/f7815818dfe8490b89e8650d70ee8fae 2023-03-31 18:54:12,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ec885308c4f5487b93003a43d84d56cd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ec885308c4f5487b93003a43d84d56cd 2023-03-31 18:54:12,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/9967b5ae49fa4ebaad3882af54dc7225 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/9967b5ae49fa4ebaad3882af54dc7225 2023-03-31 18:54:12,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/b2645fcfed0a43b3a25ee047389a2736 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/b2645fcfed0a43b3a25ee047389a2736 2023-03-31 18:54:12,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/780fa56fc9fe4b4ead088ea0e2bd5359 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/780fa56fc9fe4b4ead088ea0e2bd5359 2023-03-31 18:54:12,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/086d5c4031994d42b207f6118b6a0a55 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/086d5c4031994d42b207f6118b6a0a55 2023-03-31 18:54:12,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eb289c72dc354401accc57c8bcd73260 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eb289c72dc354401accc57c8bcd73260 2023-03-31 18:54:12,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ee3fd02fbd324d63a599b7301fa0c157 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/ee3fd02fbd324d63a599b7301fa0c157 2023-03-31 18:54:12,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/db4f92f213a34a279b4ef7db155ef910 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/db4f92f213a34a279b4ef7db155ef910 2023-03-31 18:54:12,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d9179821e57f448c95eac24794299919 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d9179821e57f448c95eac24794299919 2023-03-31 18:54:12,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/7a02aad6ac5d49f3b389d23c75e51cd7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/7a02aad6ac5d49f3b389d23c75e51cd7 2023-03-31 18:54:12,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/c2fdef4056a044fbae8e55e26f9d42c5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/c2fdef4056a044fbae8e55e26f9d42c5 2023-03-31 18:54:12,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/2b1470cdf7774b9d84325b07f5081ee0 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/2b1470cdf7774b9d84325b07f5081ee0 2023-03-31 18:54:12,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/341488f66c914264a0b0c7db85ccddc9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/341488f66c914264a0b0c7db85ccddc9 2023-03-31 18:54:12,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/3f41cf23acf243eb82bc40672031b81f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/3f41cf23acf243eb82bc40672031b81f 2023-03-31 18:54:12,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eff3e3b8a5c4454a9d4c0e3753b3f9d9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/eff3e3b8a5c4454a9d4c0e3753b3f9d9 2023-03-31 18:54:12,542 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/0b1bd8f8374d417a8eedc1944892bfc3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/0b1bd8f8374d417a8eedc1944892bfc3 2023-03-31 18:54:12,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1ccc0e0bd7a0417fbd6acc7ff72c8a50 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1ccc0e0bd7a0417fbd6acc7ff72c8a50 2023-03-31 18:54:12,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8ccb2814916d43c2bd17bce2f09bd2e5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/8ccb2814916d43c2bd17bce2f09bd2e5 2023-03-31 18:54:12,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d911784f0f134f52ab409102038851c0 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/d911784f0f134f52ab409102038851c0 2023-03-31 18:54:12,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1dd86145b5e34626a98af69a860f3ec6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/1dd86145b5e34626a98af69a860f3ec6 2023-03-31 18:54:12,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/recovered.edits/1096.seqid, newMaxSeqId=1096, maxSeqId=1 2023-03-31 18:54:12,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134. 2023-03-31 18:54:12,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 087bf54664c6a240c659c71fb3782134: 2023-03-31 18:54:12,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 087bf54664c6a240c659c71fb3782134 2023-03-31 18:54:12,554 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=087bf54664c6a240c659c71fb3782134, regionState=CLOSED 2023-03-31 18:54:12,554 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288852554"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288852554"}]},"ts":"1680288852554"} 2023-03-31 18:54:12,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-31 18:54:12,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 087bf54664c6a240c659c71fb3782134, server=jenkins-hbase4.apache.org,34951,1680288816339 in 1.5910 sec 2023-03-31 18:54:12,558 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-31 18:54:12,558 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=087bf54664c6a240c659c71fb3782134, UNASSIGN in 1.5960 sec 2023-03-31 18:54:12,559 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288852559"}]},"ts":"1680288852559"} 2023-03-31 18:54:12,560 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 18:54:12,561 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 18:54:12,563 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6130 sec 2023-03-31 18:54:13,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 18:54:13,062 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-31 18:54:13,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 18:54:13,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:13,075 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:13,077 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:13,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-31 18:54:13,082 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134 2023-03-31 18:54:13,086 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/recovered.edits] 2023-03-31 18:54:13,089 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/186d423d272d407c80a6c155778cec92 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/186d423d272d407c80a6c155778cec92 2023-03-31 18:54:13,090 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/c99f36df51a34b98929ea19894c3b6ba to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/c99f36df51a34b98929ea19894c3b6ba 2023-03-31 18:54:13,092 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/f11fe0b5dab045dc9d589919a6353a7e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/A/f11fe0b5dab045dc9d589919a6353a7e 2023-03-31 18:54:13,095 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/5bf5f5cd9ecf46c4b03302a2ad07aa83 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/5bf5f5cd9ecf46c4b03302a2ad07aa83 2023-03-31 18:54:13,096 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/c47a2c5021e64022bb29633c58ffc9fe to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/c47a2c5021e64022bb29633c58ffc9fe 2023-03-31 18:54:13,097 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/d41905b41465463d8427605da8af3eb0 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/B/d41905b41465463d8427605da8af3eb0 2023-03-31 18:54:13,099 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/4b1718dcd67041af9f7e359301f03fe1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/4b1718dcd67041af9f7e359301f03fe1 2023-03-31 18:54:13,101 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/5cc793d25b9b492d9795cd3ec062b201 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/5cc793d25b9b492d9795cd3ec062b201 2023-03-31 18:54:13,102 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/88a4015466f241339e58d698701194bd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/C/88a4015466f241339e58d698701194bd 2023-03-31 18:54:13,105 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/recovered.edits/1096.seqid to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134/recovered.edits/1096.seqid 2023-03-31 18:54:13,106 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/087bf54664c6a240c659c71fb3782134 2023-03-31 18:54:13,106 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:54:13,108 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:13,128 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 18:54:13,170 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 18:54:13,172 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:13,172 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 18:54:13,172 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680288853172"}]},"ts":"9223372036854775807"} 2023-03-31 18:54:13,174 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 18:54:13,174 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 087bf54664c6a240c659c71fb3782134, NAME => 'TestAcidGuarantees,,1680288819387.087bf54664c6a240c659c71fb3782134.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 18:54:13,174 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 18:54:13,174 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680288853174"}]},"ts":"9223372036854775807"} 2023-03-31 18:54:13,176 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 18:54:13,178 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:13,179 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 110 msec 2023-03-31 18:54:13,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-31 18:54:13,182 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-31 18:54:13,199 INFO [Listener at localhost/43327] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=231 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:34951-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1321808472_15 at /127.0.0.1:52020 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x787d1ab5-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x787d1ab5-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x787d1ab5-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x787d1ab5-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=515 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=344 (was 312) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3803 (was 4564) 2023-03-31 18:54:13,210 INFO [Listener at localhost/43327] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231, OpenFileDescriptor=515, MaxFileDescriptor=60000, SystemLoadAverage=344, ProcessCount=169, AvailableMemoryMB=3803 2023-03-31 18:54:13,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 18:54:13,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 18:54:13,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:13,215 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 18:54:13,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-31 18:54:13,216 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 18:54:13,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 18:54:13,217 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:13,218 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129 empty. 2023-03-31 18:54:13,218 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:13,218 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:54:13,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 18:54:13,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 18:54:13,627 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 18:54:13,628 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8fda10a48807d71ff04525ce9d36a129, NAME => 'TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp 2023-03-31 18:54:13,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 18:54:14,035 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:54:14,035 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8fda10a48807d71ff04525ce9d36a129, disabling compactions & flushes 2023-03-31 18:54:14,035 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:14,035 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:14,035 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. after waiting 0 ms 2023-03-31 18:54:14,035 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:14,035 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:14,035 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:14,038 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 18:54:14,039 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288854039"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288854039"}]},"ts":"1680288854039"} 2023-03-31 18:54:14,041 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 18:54:14,042 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 18:54:14,042 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288854042"}]},"ts":"1680288854042"} 2023-03-31 18:54:14,043 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 18:54:14,046 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, ASSIGN}] 2023-03-31 18:54:14,048 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, ASSIGN 2023-03-31 18:54:14,049 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34951,1680288816339; forceNewPlan=false, retain=false 2023-03-31 18:54:14,200 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=8fda10a48807d71ff04525ce9d36a129, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:14,200 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288854200"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288854200"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288854200"}]},"ts":"1680288854200"} 2023-03-31 18:54:14,202 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:54:14,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 18:54:14,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:14,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8fda10a48807d71ff04525ce9d36a129, NAME => 'TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:54:14,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:14,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:54:14,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:14,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:14,359 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:14,361 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A 2023-03-31 18:54:14,361 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A 2023-03-31 18:54:14,361 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:54:14,361 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8fda10a48807d71ff04525ce9d36a129 columnFamilyName A 2023-03-31 18:54:14,362 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(310): Store=8fda10a48807d71ff04525ce9d36a129/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:54:14,362 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:14,363 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B 2023-03-31 18:54:14,364 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B 2023-03-31 18:54:14,364 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:54:14,364 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8fda10a48807d71ff04525ce9d36a129 columnFamilyName B 2023-03-31 18:54:14,364 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(310): Store=8fda10a48807d71ff04525ce9d36a129/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:54:14,365 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:14,366 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C 2023-03-31 18:54:14,366 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C 2023-03-31 18:54:14,366 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:54:14,366 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8fda10a48807d71ff04525ce9d36a129 columnFamilyName C 2023-03-31 18:54:14,367 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(310): Store=8fda10a48807d71ff04525ce9d36a129/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:54:14,367 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:14,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:14,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:14,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:54:14,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:14,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 18:54:14,373 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8fda10a48807d71ff04525ce9d36a129; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72814103, jitterRate=0.08501468598842621}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:54:14,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:14,374 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., pid=18, masterSystemTime=1680288854354 2023-03-31 18:54:14,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:14,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:14,376 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=8fda10a48807d71ff04525ce9d36a129, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:14,376 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288854376"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680288854376"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680288854376"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680288854376"}]},"ts":"1680288854376"} 2023-03-31 18:54:14,379 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-31 18:54:14,379 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 in 175 msec 2023-03-31 18:54:14,381 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-31 18:54:14,381 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, ASSIGN in 333 msec 2023-03-31 18:54:14,381 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 18:54:14,381 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288854381"}]},"ts":"1680288854381"} 2023-03-31 18:54:14,382 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 18:54:14,385 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 18:54:14,386 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1720 sec 2023-03-31 18:54:15,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 18:54:15,321 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-31 18:54:15,322 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x7829b43e to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:15,327 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@186ae08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:15,328 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:15,332 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:15,333 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 18:54:15,338 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41860, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 18:54:15,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 18:54:15,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 18:54:15,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:15,766 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 18:54:15,768 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 18:54:15,768 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 18:54:15,772 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-31 18:54:15,784 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, REOPEN/MOVE}] 2023-03-31 18:54:15,785 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, REOPEN/MOVE 2023-03-31 18:54:15,786 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8fda10a48807d71ff04525ce9d36a129, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:15,786 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288855786"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288855786"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288855786"}]},"ts":"1680288855786"} 2023-03-31 18:54:15,787 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:54:15,939 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:15,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8fda10a48807d71ff04525ce9d36a129, disabling compactions & flushes 2023-03-31 18:54:15,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:15,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:15,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. after waiting 0 ms 2023-03-31 18:54:15,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:15,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-31 18:54:15,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:15,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:15,944 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 8fda10a48807d71ff04525ce9d36a129 to self. 2023-03-31 18:54:15,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:15,946 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8fda10a48807d71ff04525ce9d36a129, regionState=CLOSED 2023-03-31 18:54:15,947 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288855946"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288855946"}]},"ts":"1680288855946"} 2023-03-31 18:54:15,949 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-31 18:54:15,949 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 in 161 msec 2023-03-31 18:54:15,950 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34951,1680288816339; forceNewPlan=false, retain=true 2023-03-31 18:54:16,100 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8fda10a48807d71ff04525ce9d36a129, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,101 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288856100"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288856100"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288856100"}]},"ts":"1680288856100"} 2023-03-31 18:54:16,103 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:54:16,260 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8fda10a48807d71ff04525ce9d36a129, NAME => 'TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:54:16,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:16,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:54:16,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:16,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:16,263 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:16,264 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A 2023-03-31 18:54:16,264 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A 2023-03-31 18:54:16,264 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:54:16,267 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8fda10a48807d71ff04525ce9d36a129 columnFamilyName A 2023-03-31 18:54:16,269 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(310): Store=8fda10a48807d71ff04525ce9d36a129/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:54:16,269 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:16,270 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B 2023-03-31 18:54:16,270 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B 2023-03-31 18:54:16,270 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:54:16,271 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8fda10a48807d71ff04525ce9d36a129 columnFamilyName B 2023-03-31 18:54:16,271 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(310): Store=8fda10a48807d71ff04525ce9d36a129/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:54:16,271 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:16,272 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C 2023-03-31 18:54:16,272 DEBUG [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C 2023-03-31 18:54:16,272 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:54:16,273 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8fda10a48807d71ff04525ce9d36a129 columnFamilyName C 2023-03-31 18:54:16,273 INFO [StoreOpener-8fda10a48807d71ff04525ce9d36a129-1] regionserver.HStore(310): Store=8fda10a48807d71ff04525ce9d36a129/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:54:16,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:16,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:16,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:54:16,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:16,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8fda10a48807d71ff04525ce9d36a129; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73329785, jitterRate=0.09269894659519196}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:54:16,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:16,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., pid=23, masterSystemTime=1680288856256 2023-03-31 18:54:16,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,281 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8fda10a48807d71ff04525ce9d36a129, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,282 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288856281"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680288856281"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680288856281"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680288856281"}]},"ts":"1680288856281"} 2023-03-31 18:54:16,284 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-31 18:54:16,284 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 in 180 msec 2023-03-31 18:54:16,285 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-31 18:54:16,285 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, REOPEN/MOVE in 500 msec 2023-03-31 18:54:16,288 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-31 18:54:16,288 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 515 msec 2023-03-31 18:54:16,289 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 940 msec 2023-03-31 18:54:16,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-31 18:54:16,298 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x21281a37 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:16,302 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55cbb115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:16,303 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:16,307 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:16,307 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:16,311 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:16,311 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:16,316 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:16,316 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:16,319 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:16,320 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:16,324 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:16,324 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:16,328 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:16,328 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:16,332 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:16,332 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x2efdb02f to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:16,336 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@92bed09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:16,342 DEBUG [hconnection-0x7a22c5c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:16,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:16,344 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:16,344 DEBUG [hconnection-0x53b42b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:16,346 DEBUG [hconnection-0x61688c4c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:16,349 DEBUG [hconnection-0x265d86a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:16,349 DEBUG [hconnection-0x185c9a84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:16,350 DEBUG [hconnection-0x17dcca08-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:16,350 DEBUG [hconnection-0xd040231-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:16,350 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58396, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:16,353 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:16,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:16,353 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58412, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:16,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:16,353 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:16,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:16,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:16,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:16,354 DEBUG [hconnection-0x5dcd6e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:16,354 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58424, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:16,354 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,354 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,356 DEBUG [hconnection-0x62497345-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:16,359 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:16,359 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,359 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58438, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:16,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:16,359 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,359 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,361 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:16,361 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58418, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:16,364 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58450, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:16,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:16,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:16,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:16,371 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:54:16,371 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:16,371 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:16,371 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:16,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:16,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:16,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:16,372 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:16,372 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:16,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:16,372 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:16,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:16,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:16,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:16,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:16,372 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:16,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:16,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:16,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:16,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f27221[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:16,379 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:16,379 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,380 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,380 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:16,380 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:16,380 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:16,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:16,384 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:16,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:16,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:16,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:16,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:16,388 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:16,388 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:16,389 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:16,389 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:16,389 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:16,397 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,398 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:16,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:16,398 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,399 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:16,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:16,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:16,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,407 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,407 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:16,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:16,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:16,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,407 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:16,408 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288916401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288856387, max timestamp=1680288856404Num uniques 859; 2023-03-31 18:54:16,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:54:16,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288856387, max timestamp=1680288856404Num uniques 859; 2023-03-31 18:54:16,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:54:16,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288916402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288856387, max timestamp=1680288856404Num uniques 859; 2023-03-31 18:54:16,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 18:54:16,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:16,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58418 deadline: 1680288916438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58424 deadline: 1680288916444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58412 deadline: 1680288916450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,473 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331a64807842b514da8b5534840baf63389 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a64807842b514da8b5534840baf63389 2023-03-31 18:54:16,508 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:16,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:16,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288916510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288916514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:16,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:16,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:16,536 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:16,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:16,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:16,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:16,537 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,537 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,539 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:16,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:16,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:16,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:16,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:16,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:16,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:16,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:16,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:16,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:16,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58418 deadline: 1680288916557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,557 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:16,557 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:16,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:16,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:16,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,559 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:16,559 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:16,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58412 deadline: 1680288916554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,561 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,561 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58424 deadline: 1680288916558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b29784c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:16,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:16,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,564 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:16,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:16,566 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,567 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:16,567 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:16,567 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:16,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:16,570 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:16,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:16,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:16,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:16,570 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:16,574 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,574 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,574 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,574 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:16,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,576 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:16,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:16,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:16,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:16,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:16,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:16,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:16,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:16,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:16,581 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:16,581 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:16,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:16,681 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:16,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:16,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:16,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:16,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:16,691 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:16,691 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:16,691 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:16,691 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:16,691 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,693 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:16,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:16,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:16,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:16,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:16,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:16,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:16,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:16,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:16,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:16,697 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,697 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:16,697 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:16,697 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,698 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:16,698 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:16,698 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,698 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:16,698 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:16,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:16,703 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:16,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@795e7944[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:16,708 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:16,708 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:16,711 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,711 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:16,712 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:16,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:16,715 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:16,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:16,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:16,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:16,715 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:16,717 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,717 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:16,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,717 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:16,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288916714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,719 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288916716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,724 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:16,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,738 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:16,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:16,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:16,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:16,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:16,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:16,738 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:16,739 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:16,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:16,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58418 deadline: 1680288916760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58412 deadline: 1680288916762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:16,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58424 deadline: 1680288916766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,839 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:16,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:16,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:16,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:16,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:16,847 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:16,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:16,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:16,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:16,848 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,848 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,849 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,849 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:16,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:16,849 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,850 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:16,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:16,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:16,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:16,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:16,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:16,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:16,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:16,869 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,869 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:16,869 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:16,869 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:16,869 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:16,869 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:16,869 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,869 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:16,869 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:16,877 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,878 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,879 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,879 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:16,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a4b199d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:16,884 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:16,885 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,886 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,886 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:16,886 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:16,886 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:16,887 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,887 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:16,889 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:16,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:16,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:16,893 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:16,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:16,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:16,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:16,893 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:16,897 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:16,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:16,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:16,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:16,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:16,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:16,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:16,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:16,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:16,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:16,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:16,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:16,905 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:16,905 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:16,907 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:16,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:16,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:16,912 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/9202414471d3472286be797514a9a738 2023-03-31 18:54:16,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/a660aa2deda14b6d9be62e9d440ab241 2023-03-31 18:54:17,007 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:17,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:17,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:17,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:17,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:17,015 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:17,015 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,015 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:17,015 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:17,016 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,016 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,017 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,017 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,018 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,018 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:17,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288917021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288917023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:17,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:17,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:17,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:17,035 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,035 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:17,035 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:17,035 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:17,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:17,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:17,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:17,038 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:17,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b99118f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:17,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:17,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,043 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,043 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:17,043 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:17,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,043 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:17,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:17,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:17,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:17,048 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:17,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:17,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:17,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:17,048 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:17,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,057 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:17,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:17,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,058 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,058 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:17,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58418 deadline: 1680288917062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58412 deadline: 1680288917067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58424 deadline: 1680288917068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,158 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:17,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:17,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:17,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:17,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:17,170 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:17,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:17,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:17,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:17,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:17,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:17,180 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,180 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:17,180 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:17,180 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:17,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:17,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:17,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,183 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,183 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:17,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:17,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ba2e5a8[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:17,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:17,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,186 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,186 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:17,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:17,191 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:17,194 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:17,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:17,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:17,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:17,195 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:17,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,198 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,199 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:17,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:17,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,202 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:17,202 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:17,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:17,303 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:17,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:17,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:17,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:17,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:17,307 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:17,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:17,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:17,308 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,308 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:17,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:17,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:17,313 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,313 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:17,313 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:17,313 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:17,316 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,317 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,317 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:17,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:17,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:17,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61bc35c1[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:17,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,319 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:17,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:17,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:17,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:17,321 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:17,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:17,325 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:17,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:17,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:17,326 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:17,326 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:17,328 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,328 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,330 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,332 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:17,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,341 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:17,341 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:17,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:17,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/1d3cd3decac7496c82d32488a1aebf4a 2023-03-31 18:54:17,364 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/9202414471d3472286be797514a9a738 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9202414471d3472286be797514a9a738 2023-03-31 18:54:17,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9202414471d3472286be797514a9a738, entries=200, sequenceid=16, filesize=31.9 K 2023-03-31 18:54:17,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/a660aa2deda14b6d9be62e9d440ab241 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/a660aa2deda14b6d9be62e9d440ab241 2023-03-31 18:54:17,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/a660aa2deda14b6d9be62e9d440ab241, entries=150, sequenceid=16, filesize=11.6 K 2023-03-31 18:54:17,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/1d3cd3decac7496c82d32488a1aebf4a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/1d3cd3decac7496c82d32488a1aebf4a 2023-03-31 18:54:17,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/1d3cd3decac7496c82d32488a1aebf4a, entries=150, sequenceid=16, filesize=11.6 K 2023-03-31 18:54:17,405 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=124.84 KB/127835 for 8fda10a48807d71ff04525ce9d36a129 in 1034ms, sequenceid=16, compaction requested=false 2023-03-31 18:54:17,405 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-31 18:54:17,408 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:17,441 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:17,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:17,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:17,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:17,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:17,451 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:17,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:17,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:17,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,454 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,454 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:17,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:17,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:17,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,456 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:17,456 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:17,456 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:17,456 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,456 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:17,457 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=124.84 KB heapSize=354.38 KB 2023-03-31 18:54:17,457 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:17,457 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:17,457 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:17,457 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:17,457 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:17,457 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:17,509 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303311f68badeabd645568e0a18400faa71dd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311f68badeabd645568e0a18400faa71dd 2023-03-31 18:54:17,519 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/008ed16571eb439f81b063e435f98a00 2023-03-31 18:54:17,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:17,528 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:17,536 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/17e0e270cf244e70837e6fd067a8a818 2023-03-31 18:54:17,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288917543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288917545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,558 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=39.86 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/2753d0ca6e74429bb48a5627eead58c1 2023-03-31 18:54:17,563 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/008ed16571eb439f81b063e435f98a00 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/008ed16571eb439f81b063e435f98a00 2023-03-31 18:54:17,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58418 deadline: 1680288917567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,568 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/008ed16571eb439f81b063e435f98a00, entries=150, sequenceid=51, filesize=25.1 K 2023-03-31 18:54:17,569 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/17e0e270cf244e70837e6fd067a8a818 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/17e0e270cf244e70837e6fd067a8a818 2023-03-31 18:54:17,574 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/17e0e270cf244e70837e6fd067a8a818, entries=150, sequenceid=51, filesize=11.6 K 2023-03-31 18:54:17,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,576 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/2753d0ca6e74429bb48a5627eead58c1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2753d0ca6e74429bb48a5627eead58c1 2023-03-31 18:54:17,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58412 deadline: 1680288917571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58424 deadline: 1680288917580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,607 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2753d0ca6e74429bb48a5627eead58c1, entries=150, sequenceid=51, filesize=11.6 K 2023-03-31 18:54:17,608 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~124.84 KB/127835, heapSize ~354.34 KB/362840, currentSize=53.67 KB/54960 for 8fda10a48807d71ff04525ce9d36a129 in 151ms, sequenceid=51, compaction requested=false 2023-03-31 18:54:17,611 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:17,611 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:17,612 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,613 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,613 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:17,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33e94f9a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:17,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:17,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,617 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:17,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:17,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:17,618 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,618 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:17,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:17,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,619 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:17,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:17,623 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:17,623 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:17,623 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:17,623 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:17,623 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:17,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,629 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:17,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:17,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,632 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,632 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:17,633 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:17,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:17,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:17,649 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:54:17,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:17,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:17,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:17,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:17,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:17,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:17,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288917687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288917687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,733 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:17,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:17,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:17,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:17,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:17,737 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:17,737 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,737 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:17,737 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:17,737 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,737 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,739 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,739 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,739 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,739 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:17,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:17,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:17,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:17,744 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,744 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:17,744 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:17,744 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:17,747 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:17,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@400f2501[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:17,749 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:17,749 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,750 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:17,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:17,753 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,753 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,753 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,753 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:17,757 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:17,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:17,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:17,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:17,757 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:17,759 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,759 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,759 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,760 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,761 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,763 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,763 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,763 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,763 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:17,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:17,763 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,763 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,764 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,764 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:17,764 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:17,764 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,764 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,764 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:17,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288917790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288917790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,864 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:17,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:17,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:17,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:17,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:17,867 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:17,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:17,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:17,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,869 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,869 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,870 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,870 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:17,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:17,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:17,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:17,871 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:17,872 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:17,872 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:17,872 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:17,879 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,879 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,879 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,879 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:17,881 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:17,881 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7c070d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:17,882 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,882 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:17,882 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:17,882 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:17,882 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,882 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:17,883 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,884 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,884 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,884 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:17,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:17,886 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:17,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:17,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:17,887 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:17,887 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:17,888 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:17,889 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,889 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,889 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,897 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:17,897 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:17,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:17,897 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:17,897 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,897 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:17,897 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,897 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:17,898 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:17,898 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:17,897 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:17,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:17,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:17,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:17,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:17,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288917992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:17,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288917993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:17,998 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:17,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:18,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:18,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:18,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:18,002 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:18,002 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,002 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:18,002 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:18,003 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,003 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,004 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,004 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:18,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:18,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:18,006 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:18,006 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,006 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:18,011 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:18,012 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,012 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:18,012 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:18,012 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:18,014 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,014 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:18,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cee3f7e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:18,017 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:18,017 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,020 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:18,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:18,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:18,026 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:18,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:18,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:18,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:18,026 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:18,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,030 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,032 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,032 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,032 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:18,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:18,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,032 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,032 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,032 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:18,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,034 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:18,034 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:18,093 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033187b9d766d9e148de8a5048b5f035bbd5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033187b9d766d9e148de8a5048b5f035bbd5 2023-03-31 18:54:18,106 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 18:54:18,134 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:18,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:18,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:18,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:18,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:18,147 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:18,147 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,147 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:18,147 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:18,147 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,147 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,149 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,149 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:18,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:18,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:18,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:18,193 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:18,196 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,196 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:18,196 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:18,196 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,201 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:18,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@458232e6[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:18,203 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:18,203 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,204 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,204 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:18,205 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:18,207 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:18,207 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,207 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:18,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:18,210 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:18,210 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:18,210 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:18,210 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:18,210 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:18,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,215 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,217 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:18,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:18,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:18,218 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:18,218 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:18,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:18,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288918295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:18,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288918296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,318 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:18,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:18,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:18,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:18,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:18,322 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:18,322 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,322 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:18,322 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:18,322 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,322 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,325 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:18,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:18,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:18,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:18,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:18,327 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,327 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:18,327 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:18,327 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:18,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,331 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,331 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,331 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:18,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c9c207a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:18,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:18,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,335 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:18,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:18,337 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:18,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:18,340 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:18,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:18,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:18,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:18,341 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:18,343 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,343 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,343 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,344 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,350 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,350 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,350 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:18,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:18,350 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,350 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,350 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,350 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,350 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:18,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,350 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,450 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:18,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:18,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:18,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:18,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:18,456 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:18,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:18,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:18,457 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,457 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,459 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:18,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:18,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:18,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:18,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:18,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:18,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:18,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:18,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:18,469 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:18,469 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:18,473 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,473 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,473 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,473 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:18,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2084148b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:18,475 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:18,475 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,477 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,477 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:18,477 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:18,477 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:18,477 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,477 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:18,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:18,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,480 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:18,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:18,483 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:18,483 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:18,483 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:18,483 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:18,483 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:18,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,485 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,485 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,487 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,489 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:18,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:18,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:18,492 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:18,492 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:18,492 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,512 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=64, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/0148ddcd4348443fb9eea59ff71842fc 2023-03-31 18:54:18,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:18,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58418 deadline: 1680288918574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:18,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58412 deadline: 1680288918581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:18,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58424 deadline: 1680288918589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,592 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:18,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:18,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:18,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:18,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:18,596 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:18,596 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,596 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:18,596 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:18,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,598 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,599 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,599 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:18,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:18,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:18,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:18,600 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:18,600 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,600 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:18,600 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:18,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:18,601 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,601 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:18,601 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:18,601 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:18,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:18,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:18,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:18,604 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,604 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,604 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,604 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:18,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ddcda2a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:18,606 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:18,606 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,607 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,608 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,608 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:18,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:18,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:18,608 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:18,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:18,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:18,611 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:18,615 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:18,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:18,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:18,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:18,615 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:18,617 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,617 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,617 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,618 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,621 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,624 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,625 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:18,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:18,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:18,625 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:18,625 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,626 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,626 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,726 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:18,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:18,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:18,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:18,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:18,748 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:18,748 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,748 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:18,748 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:18,749 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,749 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,751 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:18,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:18,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:18,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:18,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:18,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:18,756 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,756 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:18,756 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:18,756 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:18,759 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:18,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@243fd088[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:18,761 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:18,761 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,764 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,764 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:18,764 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:18,764 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:18,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:18,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:18,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:18,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:18,771 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:18,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:18,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:18,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:18,772 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:18,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,780 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:18,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:18,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:18,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,781 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:18,781 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:18,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:18,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288918797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:18,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288918799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,881 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:18,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:18,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:18,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:18,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:18,908 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:18,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:18,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:18,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,910 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,910 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,911 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,911 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:18,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:18,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:18,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:18,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:18,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:18,917 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,917 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:18,917 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:18,917 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:18,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:18,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:18,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:18,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,920 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:18,923 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:18,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3943af[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:18,923 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,927 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,927 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:18,927 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:18,927 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:18,927 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,927 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:18,928 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:18,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:18,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:18,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:18,932 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:18,932 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:18,932 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:18,932 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:18,932 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:18,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:18,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,935 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,938 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:18,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/22921ed69f054213955be3d2bca6dbf4 2023-03-31 18:54:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:18,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:18,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:18,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:18,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:18,943 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:18,943 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:18,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:18,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:18,944 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:18,944 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:18,944 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:18,944 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:18,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:18,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:18,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/6d585de531f74c0197253ec47aff3f15 2023-03-31 18:54:18,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/0148ddcd4348443fb9eea59ff71842fc as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0148ddcd4348443fb9eea59ff71842fc 2023-03-31 18:54:18,977 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0148ddcd4348443fb9eea59ff71842fc, entries=200, sequenceid=64, filesize=31.9 K 2023-03-31 18:54:18,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/22921ed69f054213955be3d2bca6dbf4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/22921ed69f054213955be3d2bca6dbf4 2023-03-31 18:54:18,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/22921ed69f054213955be3d2bca6dbf4, entries=150, sequenceid=64, filesize=11.6 K 2023-03-31 18:54:18,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/6d585de531f74c0197253ec47aff3f15 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d585de531f74c0197253ec47aff3f15 2023-03-31 18:54:18,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d585de531f74c0197253ec47aff3f15, entries=150, sequenceid=64, filesize=11.6 K 2023-03-31 18:54:18,993 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 8fda10a48807d71ff04525ce9d36a129 in 1344ms, sequenceid=64, compaction requested=true 2023-03-31 18:54:18,993 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:18,993 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:18,993 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:54:18,994 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:54:18,993 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:18,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:18,997 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91118 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:54:18,997 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/A is initiating minor compaction (all files) 2023-03-31 18:54:18,997 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:54:18,997 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/A in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,998 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/B is initiating minor compaction (all files) 2023-03-31 18:54:18,998 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/B in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:18,998 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9202414471d3472286be797514a9a738, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/008ed16571eb439f81b063e435f98a00, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0148ddcd4348443fb9eea59ff71842fc] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=89.0 K 2023-03-31 18:54:18,998 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/a660aa2deda14b6d9be62e9d440ab241, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/17e0e270cf244e70837e6fd067a8a818, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/22921ed69f054213955be3d2bca6dbf4] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=34.8 K 2023-03-31 18:54:18,998 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9202414471d3472286be797514a9a738, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680288856367 2023-03-31 18:54:18,998 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a660aa2deda14b6d9be62e9d440ab241, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680288856367 2023-03-31 18:54:18,999 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 008ed16571eb439f81b063e435f98a00, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680288856435 2023-03-31 18:54:18,999 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 17e0e270cf244e70837e6fd067a8a818, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680288856435 2023-03-31 18:54:18,999 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 0148ddcd4348443fb9eea59ff71842fc, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1680288857535 2023-03-31 18:54:18,999 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 22921ed69f054213955be3d2bca6dbf4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1680288857535 2023-03-31 18:54:19,009 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#B#compaction#112 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:19,033 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#A#compaction#113 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:19,045 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:19,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:19,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:19,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:19,065 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:19,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:19,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:19,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,066 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,067 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,067 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:19,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:19,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:19,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:19,072 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,073 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:19,073 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:54:19,073 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:19,073 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:19,074 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:19,074 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:19,074 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:19,074 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:19,113 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331e9d6a59e3053488992d1cc838fc56f30 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e9d6a59e3053488992d1cc838fc56f30 2023-03-31 18:54:19,129 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/a513c1c7ec9b487097524285209ef778 2023-03-31 18:54:19,153 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/55dde17050b64f2aad0bf36c57286f3c 2023-03-31 18:54:19,170 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/065a3d45db574555adc008f6ac2b0b74 2023-03-31 18:54:19,177 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/a513c1c7ec9b487097524285209ef778 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a513c1c7ec9b487097524285209ef778 2023-03-31 18:54:19,183 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a513c1c7ec9b487097524285209ef778, entries=150, sequenceid=84, filesize=25.1 K 2023-03-31 18:54:19,184 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/55dde17050b64f2aad0bf36c57286f3c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/55dde17050b64f2aad0bf36c57286f3c 2023-03-31 18:54:19,189 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/55dde17050b64f2aad0bf36c57286f3c, entries=150, sequenceid=84, filesize=11.6 K 2023-03-31 18:54:19,190 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/065a3d45db574555adc008f6ac2b0b74 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/065a3d45db574555adc008f6ac2b0b74 2023-03-31 18:54:19,210 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/065a3d45db574555adc008f6ac2b0b74, entries=150, sequenceid=84, filesize=11.6 K 2023-03-31 18:54:19,211 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 8fda10a48807d71ff04525ce9d36a129 in 138ms, sequenceid=84, compaction requested=true 2023-03-31 18:54:19,211 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:19,211 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:19,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:19,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7870b045[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:19,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:19,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:19,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:19,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:19,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:19,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:19,219 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:19,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:19,223 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:19,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:19,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:19,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:19,223 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:19,225 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,225 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,225 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,226 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,227 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,232 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,232 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:19,232 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,232 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:19,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:19,233 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,233 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,233 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,233 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:19,233 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:19,333 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:19,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:19,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:19,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:19,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:19,352 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:19,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:19,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:19,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,355 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,359 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,359 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,359 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:19,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:19,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:19,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:19,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,367 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:19,386 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:19,386 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,386 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:19,386 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:19,386 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:19,386 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,386 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:19,386 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:19,386 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:19,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:19,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,389 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,389 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:19,391 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:19,391 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21f0becf[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:19,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,392 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:19,392 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:19,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:19,393 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:19,394 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,394 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:19,394 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,394 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:19,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:19,397 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:19,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:19,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:19,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:19,397 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:19,401 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,401 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,401 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,402 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,403 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:19,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:19,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,407 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,407 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,407 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,407 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:19,407 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:19,407 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,407 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,407 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:19,456 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/36cb8fab67314bf9a44b87f50b5847de as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/36cb8fab67314bf9a44b87f50b5847de 2023-03-31 18:54:19,466 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/B of 8fda10a48807d71ff04525ce9d36a129 into 36cb8fab67314bf9a44b87f50b5847de(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:19,466 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:19,466 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/B, priority=13, startTime=1680288858993; duration=0sec 2023-03-31 18:54:19,466 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:19,467 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:54:19,468 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:54:19,468 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/C is initiating minor compaction (all files) 2023-03-31 18:54:19,469 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/C in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,469 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/1d3cd3decac7496c82d32488a1aebf4a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2753d0ca6e74429bb48a5627eead58c1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d585de531f74c0197253ec47aff3f15, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/065a3d45db574555adc008f6ac2b0b74] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=46.4 K 2023-03-31 18:54:19,469 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 1d3cd3decac7496c82d32488a1aebf4a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680288856367 2023-03-31 18:54:19,470 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2753d0ca6e74429bb48a5627eead58c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680288856435 2023-03-31 18:54:19,470 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 6d585de531f74c0197253ec47aff3f15, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1680288857535 2023-03-31 18:54:19,471 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 065a3d45db574555adc008f6ac2b0b74, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680288857657 2023-03-31 18:54:19,486 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#C#compaction#117 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:19,508 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:19,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:19,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:19,519 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:19,519 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,519 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:19,520 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:19,520 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,520 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,522 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/be0494c20f1243e3b57287a35c39c561 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/be0494c20f1243e3b57287a35c39c561 2023-03-31 18:54:19,523 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,523 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,524 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,524 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:19,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:19,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:19,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:19,532 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,532 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:19,532 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:19,533 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:19,533 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/C of 8fda10a48807d71ff04525ce9d36a129 into be0494c20f1243e3b57287a35c39c561(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:19,533 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:19,533 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/C, priority=12, startTime=1680288858994; duration=0sec 2023-03-31 18:54:19,534 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:19,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,535 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b1181e9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:19,539 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:19,539 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,543 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,543 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,543 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:19,543 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:19,543 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:19,544 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,545 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:19,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:19,548 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:19,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:19,549 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:19,549 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:19,549 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:19,553 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,553 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,553 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:19,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:19,560 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:19,560 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:19,560 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:19,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,561 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,561 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,661 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:19,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:19,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:19,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:19,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:19,672 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:19,672 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,672 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:19,672 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:19,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,674 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,674 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,674 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,674 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:19,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:19,675 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:19,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:19,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:19,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:19,683 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,683 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,683 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,683 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e50811f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:19,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:19,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,686 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:19,687 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,687 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:19,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:19,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:19,689 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:19,692 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:19,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:19,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:19,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:19,692 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:19,694 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,694 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,694 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,697 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,700 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,700 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,700 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:19,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:19,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:19,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,700 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:19,700 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:19,700 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,801 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:19,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:19,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:19,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:19,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:19,805 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:19,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:19,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:19,806 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,806 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,807 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,807 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,808 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,808 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:19,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:19,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:19,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:19,810 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,810 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:19,812 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,813 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:19,813 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:19,813 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,817 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:19,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@635ff808[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:19,819 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:19,819 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,820 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,820 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:19,821 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,821 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:19,822 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,822 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:19,822 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,822 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:19,825 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:19,825 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:19,825 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:19,825 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:19,825 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:19,827 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,827 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,827 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,833 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,833 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,833 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,833 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,833 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,833 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,833 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,833 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:19,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:19,833 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,833 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,834 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:19,834 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:19,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:19,868 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:54:19,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:19,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:19,870 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:19,870 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:19,870 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:19,870 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:19,922 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/4f2ab3c174754301b1503c833e32b048 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/4f2ab3c174754301b1503c833e32b048 2023-03-31 18:54:19,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288859914, max timestamp=1680288859925Num uniques 859; 2023-03-31 18:54:19,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:54:19,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288859914, max timestamp=1680288859925Num uniques 859; 2023-03-31 18:54:19,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:54:19,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288859914, max timestamp=1680288859925Num uniques 859; 2023-03-31 18:54:19,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:54:19,932 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/A of 8fda10a48807d71ff04525ce9d36a129 into 4f2ab3c174754301b1503c833e32b048(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:19,933 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:19,933 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/A, priority=13, startTime=1680288858993; duration=0sec 2023-03-31 18:54:19,933 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:19,934 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:19,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:19,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:19,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:19,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:19,940 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:19,940 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,940 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:19,940 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:19,940 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,940 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:19,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:19,943 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:19,943 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:19,943 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:19,948 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,948 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:19,948 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:19,948 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:19,950 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:19,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:19,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477d346a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:19,953 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,954 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:19,956 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:19,956 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,956 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:19,959 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:19,959 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:19,959 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:19,959 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:19,959 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:19,961 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,961 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:19,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,961 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:19,962 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,963 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:19,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:19,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288919962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,968 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:19,969 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:19,968 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:19,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:19,969 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:19,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:19,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:19,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288919964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:19,969 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:19,969 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:19,969 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:19,969 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:19,970 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:19,970 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,970 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:19,970 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:20,069 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:20,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288920066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:20,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:20,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288920070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:20,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:20,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:20,074 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:20,074 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,074 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:20,074 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:20,074 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,074 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,076 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:20,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:20,078 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,078 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:20,078 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:20,078 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:20,079 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,079 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,081 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,081 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:20,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3014eba5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:20,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,084 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:20,084 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:20,084 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:20,084 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,084 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:20,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:20,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:20,088 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:20,088 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:20,088 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:20,088 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:20,088 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:20,090 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,090 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,090 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,090 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,092 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,095 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,095 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,095 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:20,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:20,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:20,095 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,095 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:20,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,096 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:20,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,196 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:20,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:20,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:20,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:20,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:20,200 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:20,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:20,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:20,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:20,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:20,204 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,204 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:20,204 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:20,204 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:20,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,207 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:20,208 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:20,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a56ccc1[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:20,208 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:20,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:20,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:20,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:20,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:20,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:20,218 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:20,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:20,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:20,219 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:20,219 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:20,221 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,221 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,221 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,224 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,226 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,226 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,226 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,226 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,226 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,226 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,227 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,227 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,227 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,227 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:20,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:20,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,227 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:20,227 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:20,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:20,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288920271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:20,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288920273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,318 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033136bf2e8eab8b45d4bb8e630ead2ea87d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033136bf2e8eab8b45d4bb8e630ead2ea87d 2023-03-31 18:54:20,327 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:20,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:20,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:20,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:20,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:20,331 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:20,331 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,331 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:20,331 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:20,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,335 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:20,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:20,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:20,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:20,340 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,340 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:20,340 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:20,340 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:20,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,342 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3655ee3b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:20,344 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:20,344 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,345 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:20,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:20,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:20,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:20,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,349 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:20,349 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,349 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:20,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:20,353 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:20,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:20,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:20,353 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:20,353 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:20,356 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,356 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,356 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,357 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,360 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,366 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,366 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,366 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,366 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,366 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,366 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,366 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,366 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,366 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:20,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:20,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,367 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:20,367 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:20,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:20,467 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:20,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:20,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:20,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:20,471 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:20,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:20,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:20,472 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,472 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,473 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,473 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,473 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,473 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:20,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:20,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:20,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:20,475 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,475 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:20,475 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:20,475 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,476 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:20,476 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:20,478 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,478 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,478 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,478 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:20,480 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:20,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@182542ff[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:20,480 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,482 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:20,482 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,482 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:20,483 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,484 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,484 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:20,487 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:20,487 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:20,487 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:20,487 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:20,487 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:20,490 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,490 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,492 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,511 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:20,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:20,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,511 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:20,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:20,511 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:20,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:20,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288920573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:20,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288920578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:20,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58412 deadline: 1680288920587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,590 DEBUG [Thread-861] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:20,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:20,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58418 deadline: 1680288920592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,595 DEBUG [Thread-869] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:20,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:20,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58424 deadline: 1680288920601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,603 DEBUG [Thread-865] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:20,611 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:20,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:20,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:20,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:20,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:20,615 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:20,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:20,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:20,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,617 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:20,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:20,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:20,619 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,619 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:20,619 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:20,619 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:20,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ffbcb9d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:20,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:20,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,626 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:20,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:20,629 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:20,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:20,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:20,637 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:20,637 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:20,637 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:20,637 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:20,637 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:20,644 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,644 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,644 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,648 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,650 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:20,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:20,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:20,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,650 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:20,651 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:20,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,727 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=101, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/b0a81ab8d0a942738c5d2412916abd2a 2023-03-31 18:54:20,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/e20eaa4d880a4464a830b9151aa2ae79 2023-03-31 18:54:20,751 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:20,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:20,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:20,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:20,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:20,754 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:20,754 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,754 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:20,754 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:20,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,757 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:20,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:20,759 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,759 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:20,759 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:20,759 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:20,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,764 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,764 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:20,764 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:20,766 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13572e67[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:20,766 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,767 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:20,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:20,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:20,769 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,769 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:20,769 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,769 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:20,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:20,771 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:20,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:20,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:20,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:20,772 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:20,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,774 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,782 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,785 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,785 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,785 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,785 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,785 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,785 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,785 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,787 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:20,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:20,787 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:20,787 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:20,887 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:20,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:20,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:20,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:20,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:20,891 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:20,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:20,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:20,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,893 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:20,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:20,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:20,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:20,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:20,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:20,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:20,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:20,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:20,901 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:20,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:20,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a6a3e00[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:20,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:20,905 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:20,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:20,906 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:20,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:20,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:20,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:20,906 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:20,906 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:20,906 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,906 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:20,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:20,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,910 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:20,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:20,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:20,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:20,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:20,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:20,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:20,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:20,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:20,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:20,917 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:20,917 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:20,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:20,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:21,017 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:21,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:21,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:21,021 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:21,021 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:21,021 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:21,021 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:21,021 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,021 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:21,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:21,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:21,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:21,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:21,025 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:21,025 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:21,025 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:21,025 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:21,025 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:21,025 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:21,025 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,025 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:21,025 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:21,026 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,026 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386ab708[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:21,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:21,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,031 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:21,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:21,031 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:21,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:21,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:21,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:21,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:21,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:21,036 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:21,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:21,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:21,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:21,036 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:21,038 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,038 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:21,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:21,039 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,040 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:21,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:21,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:21,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:21,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:21,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:21,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:21,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:21,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:21,045 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:21,045 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:21,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:21,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:21,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:21,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:21,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288921080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:21,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288921082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,145 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:21,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:21,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:21,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:21,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:21,149 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:21,149 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:21,149 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:21,149 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:21,149 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/3a087997f23f4558b444b2eb1c95138d 2023-03-31 18:54:21,151 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,151 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:21,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:21,151 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,151 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:21,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:21,154 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:21,154 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:21,154 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:21,154 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:21,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:21,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:21,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:21,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:21,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/b0a81ab8d0a942738c5d2412916abd2a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/b0a81ab8d0a942738c5d2412916abd2a 2023-03-31 18:54:21,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,159 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:21,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:21,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:21,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699c34bb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:21,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:21,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/b0a81ab8d0a942738c5d2412916abd2a, entries=350, sequenceid=101, filesize=52.3 K 2023-03-31 18:54:21,164 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:21,164 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,164 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:21,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/e20eaa4d880a4464a830b9151aa2ae79 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e20eaa4d880a4464a830b9151aa2ae79 2023-03-31 18:54:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:21,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:21,166 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e20eaa4d880a4464a830b9151aa2ae79, entries=150, sequenceid=101, filesize=11.6 K 2023-03-31 18:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:21,169 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:21,169 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:21,169 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:21,169 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:21,169 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:21,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/3a087997f23f4558b444b2eb1c95138d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/3a087997f23f4558b444b2eb1c95138d 2023-03-31 18:54:21,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:21,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:21,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:21,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:21,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:21,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:21,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:21,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:21,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:21,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:21,176 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:21,176 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:21,176 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,176 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,176 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:21,176 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:21,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/3a087997f23f4558b444b2eb1c95138d, entries=150, sequenceid=101, filesize=11.6 K 2023-03-31 18:54:21,178 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8fda10a48807d71ff04525ce9d36a129 in 1310ms, sequenceid=101, compaction requested=true 2023-03-31 18:54:21,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:21,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:21,179 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:54:21,179 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:54:21,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:21,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:21,180 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105105 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:54:21,180 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/A is initiating minor compaction (all files) 2023-03-31 18:54:21,180 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/A in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:21,180 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:54:21,180 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/4f2ab3c174754301b1503c833e32b048, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a513c1c7ec9b487097524285209ef778, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/b0a81ab8d0a942738c5d2412916abd2a] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=102.6 K 2023-03-31 18:54:21,180 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/B is initiating minor compaction (all files) 2023-03-31 18:54:21,180 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/B in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:21,181 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/36cb8fab67314bf9a44b87f50b5847de, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/55dde17050b64f2aad0bf36c57286f3c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e20eaa4d880a4464a830b9151aa2ae79] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=34.9 K 2023-03-31 18:54:21,181 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 4f2ab3c174754301b1503c833e32b048, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1680288857535 2023-03-31 18:54:21,184 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 36cb8fab67314bf9a44b87f50b5847de, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1680288857535 2023-03-31 18:54:21,184 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a513c1c7ec9b487097524285209ef778, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680288857657 2023-03-31 18:54:21,184 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 55dde17050b64f2aad0bf36c57286f3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680288857657 2023-03-31 18:54:21,185 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting b0a81ab8d0a942738c5d2412916abd2a, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680288859818 2023-03-31 18:54:21,185 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e20eaa4d880a4464a830b9151aa2ae79, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680288859852 2023-03-31 18:54:21,197 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#B#compaction#122 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:21,206 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#A#compaction#121 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:21,227 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/d5e7da09a84a4811b94d27cd9dbc376b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/d5e7da09a84a4811b94d27cd9dbc376b 2023-03-31 18:54:21,234 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/B of 8fda10a48807d71ff04525ce9d36a129 into d5e7da09a84a4811b94d27cd9dbc376b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:21,234 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:21,234 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/B, priority=13, startTime=1680288861179; duration=0sec 2023-03-31 18:54:21,234 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:21,234 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 18:54:21,235 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 18:54:21,235 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 18:54:21,235 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. because compaction request was cancelled 2023-03-31 18:54:21,276 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:21,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:21,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:21,288 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:21,288 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:21,288 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:21,288 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:21,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,291 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:21,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:21,295 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:21,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:21,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:21,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:21,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:21,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:21,317 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:21,317 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:21,317 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:21,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:21,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:21,344 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:21,344 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:21,344 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-31 18:54:21,344 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:21,345 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:21,345 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:21,345 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:21,345 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:21,345 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:21,426 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033193ed7b1dbd8547f49e00f17fdd015397 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033193ed7b1dbd8547f49e00f17fdd015397 2023-03-31 18:54:21,750 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/0dea6dfbb3c848828fdf393e9f89df62 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0dea6dfbb3c848828fdf393e9f89df62 2023-03-31 18:54:21,756 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/A of 8fda10a48807d71ff04525ce9d36a129 into 0dea6dfbb3c848828fdf393e9f89df62(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:21,756 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:21,756 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/A, priority=13, startTime=1680288861179; duration=0sec 2023-03-31 18:54:21,756 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:21,840 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=138, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/1124ed502f064594823b1bfa25c9fda6 2023-03-31 18:54:22,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:22,087 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:22,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:22,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288922098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:22,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288922101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:22,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288922203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:22,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288922205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,287 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/9ff667eb30e54c54bbcbb81b12e515d4 2023-03-31 18:54:22,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:22,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288922407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:22,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288922409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,701 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/a26f40721fed449bb3906b553bee29a1 2023-03-31 18:54:22,708 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/1124ed502f064594823b1bfa25c9fda6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/1124ed502f064594823b1bfa25c9fda6 2023-03-31 18:54:22,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288922711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:22,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288922711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,713 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/1124ed502f064594823b1bfa25c9fda6, entries=150, sequenceid=138, filesize=25.3 K 2023-03-31 18:54:22,715 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/9ff667eb30e54c54bbcbb81b12e515d4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/9ff667eb30e54c54bbcbb81b12e515d4 2023-03-31 18:54:22,722 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/9ff667eb30e54c54bbcbb81b12e515d4, entries=150, sequenceid=138, filesize=11.8 K 2023-03-31 18:54:22,723 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/a26f40721fed449bb3906b553bee29a1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/a26f40721fed449bb3906b553bee29a1 2023-03-31 18:54:22,730 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/a26f40721fed449bb3906b553bee29a1, entries=150, sequenceid=138, filesize=11.8 K 2023-03-31 18:54:22,731 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 8fda10a48807d71ff04525ce9d36a129 in 1387ms, sequenceid=138, compaction requested=true 2023-03-31 18:54:22,731 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:22,731 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:22,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:22,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:22,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,732 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:22,732 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:22,735 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,735 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:22,735 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:22,735 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:22,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:22,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:22,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:22,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43120bf9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:22,739 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:22,739 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:22,742 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:22,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:22,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:22,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:22,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:22,743 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,743 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:22,744 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:22,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:22,749 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:22,749 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:22,749 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:22,749 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:22,749 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:22,751 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:22,751 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:22,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:22,752 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:22,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:22,752 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:22,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,759 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:22,759 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:22,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:22,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:22,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:22,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:22,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,760 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:22,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:22,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:22,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:22,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:22,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:22,760 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:22,760 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:22,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:22,861 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:22,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:22,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:22,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:22,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:22,864 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:22,864 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:22,864 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:22,864 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:22,865 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:22,865 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:22,866 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:22,866 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:22,866 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:22,866 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:22,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:22,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:22,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:22,868 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:22,868 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:22,868 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:54:22,869 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:22,869 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:22,869 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:22,869 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:22,869 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:22,869 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:23,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:23,214 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:23,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288863235, max timestamp=1680288863246Num uniques 859; 2023-03-31 18:54:23,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288863235, max timestamp=1680288863246Num uniques 859; 2023-03-31 18:54:23,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:23,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:23,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288863235, max timestamp=1680288863246Num uniques 859; 2023-03-31 18:54:23,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:23,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:23,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288923289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:23,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288923291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,300 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303313aee8ab940ce401ea071a22f9b97b61a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313aee8ab940ce401ea071a22f9b97b61a 2023-03-31 18:54:23,306 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/0cb5b300305c4ba6b6af87b17761a4f6 2023-03-31 18:54:23,316 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/56173d0709d7430384e102a3f2947d92 2023-03-31 18:54:23,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:23,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288923394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:23,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288923396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:23,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288923599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:23,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288923599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,728 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/0d422e4536754976ab48e09cbf35865d 2023-03-31 18:54:23,734 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/0cb5b300305c4ba6b6af87b17761a4f6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0cb5b300305c4ba6b6af87b17761a4f6 2023-03-31 18:54:23,739 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0cb5b300305c4ba6b6af87b17761a4f6, entries=150, sequenceid=150, filesize=25.3 K 2023-03-31 18:54:23,740 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/56173d0709d7430384e102a3f2947d92 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/56173d0709d7430384e102a3f2947d92 2023-03-31 18:54:23,743 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/56173d0709d7430384e102a3f2947d92, entries=150, sequenceid=150, filesize=11.8 K 2023-03-31 18:54:23,744 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/0d422e4536754976ab48e09cbf35865d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/0d422e4536754976ab48e09cbf35865d 2023-03-31 18:54:23,748 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/0d422e4536754976ab48e09cbf35865d, entries=150, sequenceid=150, filesize=11.8 K 2023-03-31 18:54:23,750 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8fda10a48807d71ff04525ce9d36a129 in 882ms, sequenceid=150, compaction requested=true 2023-03-31 18:54:23,750 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:23,750 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:23,751 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,751 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:23,752 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:23,752 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:23,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:23,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d3b7a5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:23,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:23,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:23,754 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:23,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:23,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:23,757 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:23,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:23,759 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:23,760 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:23,760 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:23,760 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:23,760 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:23,761 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:23,761 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:23,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:23,761 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:23,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:23,763 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,767 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,767 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:23,767 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:23,767 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:23,767 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:23,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:23,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:23,767 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:23,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:23,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:23,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:23,767 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:23,768 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:23,768 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:23,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:23,868 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:23,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:23,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:23,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:23,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:23,872 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:23,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:23,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:23,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:23,873 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:23,873 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:23,874 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,874 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:23,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:23,874 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,874 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:23,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:23,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:23,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:23,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:23,875 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:23,875 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:23,875 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:23,876 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:23,876 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:23,876 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-31 18:54:23,880 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:23,880 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:23,880 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:23,880 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:23,880 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:23,880 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:23,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:23,905 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:23,907 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331f10fd65bfce941e899d71c354e732ea9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f10fd65bfce941e899d71c354e732ea9 2023-03-31 18:54:23,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:23,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288923921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:23,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:23,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288923921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:24,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:24,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288924027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:24,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:24,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288924028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:24,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:24,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288924229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:24,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:24,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288924234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:24,322 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=186, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/ad788f90569242a4a12e6cc7d00da7f5 2023-03-31 18:54:24,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:24,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288924535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:24,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288924538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:24,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:24,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58418 deadline: 1680288924623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:24,625 DEBUG [Thread-869] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:24,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:24,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58412 deadline: 1680288924626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:24,629 DEBUG [Thread-861] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:24,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:24,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58424 deadline: 1680288924634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:24,636 DEBUG [Thread-865] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:24,742 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/f44c406257924ac6b077a37992146e47 2023-03-31 18:54:25,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:25,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58384 deadline: 1680288925039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:25,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58396 deadline: 1680288925045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,154 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/d55046dd82b24ace908be96a1cb1544a 2023-03-31 18:54:25,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/ad788f90569242a4a12e6cc7d00da7f5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ad788f90569242a4a12e6cc7d00da7f5 2023-03-31 18:54:25,164 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ad788f90569242a4a12e6cc7d00da7f5, entries=150, sequenceid=186, filesize=25.3 K 2023-03-31 18:54:25,165 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/f44c406257924ac6b077a37992146e47 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/f44c406257924ac6b077a37992146e47 2023-03-31 18:54:25,169 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/f44c406257924ac6b077a37992146e47, entries=150, sequenceid=186, filesize=11.8 K 2023-03-31 18:54:25,170 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/d55046dd82b24ace908be96a1cb1544a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/d55046dd82b24ace908be96a1cb1544a 2023-03-31 18:54:25,174 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/d55046dd82b24ace908be96a1cb1544a, entries=150, sequenceid=186, filesize=11.8 K 2023-03-31 18:54:25,175 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 8fda10a48807d71ff04525ce9d36a129 in 1299ms, sequenceid=186, compaction requested=true 2023-03-31 18:54:25,175 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:25,175 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:25,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:25,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:25,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:25,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:25,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@756c8140[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:25,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:25,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,187 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:25,187 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:25,187 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:25,188 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,188 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:25,189 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:25,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:25,194 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:25,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:25,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:25,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:25,194 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:25,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:25,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:25,198 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:25,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:25,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:25,204 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:25,204 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:25,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:25,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,204 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:25,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:25,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:25,304 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:25,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:25,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:25,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:25,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:25,308 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:25,308 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:25,308 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:25,308 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:25,309 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,309 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:25,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:25,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:25,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:25,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:25,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:25,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:25,311 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:25,311 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:25,311 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:25,312 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:25,312 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:25,312 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:54:25,312 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:25,312 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:25,312 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:25,312 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:25,312 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:25,312 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:25,342 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303318346391e559145588e3081ca68e6bb8f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318346391e559145588e3081ca68e6bb8f 2023-03-31 18:54:25,348 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=198, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/ee94c2a734f849b49b9f6e1adc375bcd 2023-03-31 18:54:25,766 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/7fa125b2cf87464591c6ec848e174d46 2023-03-31 18:54:25,778 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/e95fcb9d8ad94ea99bfa1fcba6e06ca5 2023-03-31 18:54:25,783 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/ee94c2a734f849b49b9f6e1adc375bcd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ee94c2a734f849b49b9f6e1adc375bcd 2023-03-31 18:54:25,787 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ee94c2a734f849b49b9f6e1adc375bcd, entries=150, sequenceid=198, filesize=25.3 K 2023-03-31 18:54:25,788 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/7fa125b2cf87464591c6ec848e174d46 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7fa125b2cf87464591c6ec848e174d46 2023-03-31 18:54:25,797 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7fa125b2cf87464591c6ec848e174d46, entries=150, sequenceid=198, filesize=11.8 K 2023-03-31 18:54:25,798 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/e95fcb9d8ad94ea99bfa1fcba6e06ca5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e95fcb9d8ad94ea99bfa1fcba6e06ca5 2023-03-31 18:54:25,804 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e95fcb9d8ad94ea99bfa1fcba6e06ca5, entries=150, sequenceid=198, filesize=11.8 K 2023-03-31 18:54:25,805 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 8fda10a48807d71ff04525ce9d36a129 in 493ms, sequenceid=198, compaction requested=true 2023-03-31 18:54:25,805 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:25,805 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:25,805 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:25,817 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:25,817 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:25,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:25,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:25,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ddd4aba[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:25,820 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:25,820 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,822 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:25,823 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,823 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:25,831 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:25,832 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:25,832 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:25,831 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:25,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:25,834 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:25,834 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:25,834 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:25,834 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:25,834 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:25,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:25,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:25,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:25,837 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,838 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:25,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:25,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:25,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:25,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:25,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:25,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,841 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,841 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:25,841 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:25,841 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,841 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:25,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:25,941 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:25,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:25,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:25,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:25,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:25,945 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:25,945 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:25,945 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:25,945 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:25,946 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,946 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,947 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:25,947 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,948 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,948 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:25,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:25,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:25,949 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:25,949 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:25,949 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:25,949 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:25,949 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:25,949 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:25,950 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:25,950 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:25,950 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:25,950 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:25,950 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:25,950 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,950 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:25,950 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,951 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:25,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:25,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec3cb3b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:25,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:25,955 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,955 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:25,957 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:25,957 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,957 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:25,961 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:25,961 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:25,961 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:25,961 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:25,961 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:25,963 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,963 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:25,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,963 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:25,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:25,963 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,965 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,974 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,974 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:25,974 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:25,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:25,974 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:25,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:25,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:25,974 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:25,974 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:25,975 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:25,975 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:25,975 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:25,975 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:25,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:25,975 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:25,975 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:26,075 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:26,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:26,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:26,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:26,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:26,079 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:26,079 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:26,079 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:26,079 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:26,079 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:26,079 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:26,081 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,081 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:26,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:26,081 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,081 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:26,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:26,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:26,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:26,083 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:26,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:26,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:26,088 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:26,088 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:26,088 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-31 18:54:26,088 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:26,089 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:26,089 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:26,089 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:26,089 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:26,089 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:26,110 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303311eec44c476474ccf92ea6e16d7703130 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311eec44c476474ccf92ea6e16d7703130 2023-03-31 18:54:26,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:26,112 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:26,119 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=204, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/070fc876f1604cbb9340607a41add227 2023-03-31 18:54:26,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680288866100, max timestamp=1680288866154Num uniques 859; 2023-03-31 18:54:26,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-31 18:54:26,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680288866100, max timestamp=1680288866154Num uniques 859; 2023-03-31 18:54:26,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-31 18:54:26,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680288866100, max timestamp=1680288866154Num uniques 859; 2023-03-31 18:54:26,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-31 18:54:26,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:26,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288926175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:26,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288926177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:26,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288926278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:26,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288926283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:26,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288926482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:26,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288926490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,542 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/ae35692b5bf44904804b697c6dba7af5 2023-03-31 18:54:26,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:26,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288926788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:26,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288926799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,954 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/e7dc4248870d4591879d3c29f828af69 2023-03-31 18:54:26,958 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/070fc876f1604cbb9340607a41add227 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/070fc876f1604cbb9340607a41add227 2023-03-31 18:54:26,962 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/070fc876f1604cbb9340607a41add227, entries=100, sequenceid=204, filesize=18.5 K 2023-03-31 18:54:26,963 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/ae35692b5bf44904804b697c6dba7af5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/ae35692b5bf44904804b697c6dba7af5 2023-03-31 18:54:26,967 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/ae35692b5bf44904804b697c6dba7af5, entries=100, sequenceid=204, filesize=9.4 K 2023-03-31 18:54:26,968 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/e7dc4248870d4591879d3c29f828af69 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e7dc4248870d4591879d3c29f828af69 2023-03-31 18:54:26,972 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e7dc4248870d4591879d3c29f828af69, entries=50, sequenceid=204, filesize=7.1 K 2023-03-31 18:54:26,973 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=187.85 KB/192360 for 8fda10a48807d71ff04525ce9d36a129 in 885ms, sequenceid=204, compaction requested=true 2023-03-31 18:54:26,973 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:26,973 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:26,976 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,976 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:26,976 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:26,976 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba83fd9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:26,978 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:26,978 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:26,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:26,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:26,979 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:26,979 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:26,979 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:26,979 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,979 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:26,981 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:26,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:26,983 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:26,983 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:26,983 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:26,983 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:26,983 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:26,985 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:26,985 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:26,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:26,985 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:26,985 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:26,987 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,990 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:26,990 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:26,990 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,990 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:26,990 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:26,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:26,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:26,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:26,990 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:26,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:26,990 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:26,990 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:26,990 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:26,990 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:26,990 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:26,990 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:26,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:27,090 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:27,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:27,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:27,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:27,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:27,094 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:27,094 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:27,094 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:27,094 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:27,095 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:27,095 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:27,096 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:27,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:27,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:27,096 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:27,096 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:27,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:27,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:27,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:27,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:27,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:27,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:27,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:27,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:27,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:27,098 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:27,098 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:27,098 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=187.85 KB heapSize=478.73 KB 2023-03-31 18:54:27,099 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:27,099 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:27,099 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:27,099 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:27,099 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:27,099 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:27,120 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303313069b4a068ed45fc961141b9412c7d23 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313069b4a068ed45fc961141b9412c7d23 2023-03-31 18:54:27,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:27,296 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:27,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:27,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288927301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:27,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:27,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288927305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:27,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:27,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288927403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:27,526 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=235, memsize=62.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/c7cbcabb2f814ae2b21e67660493f0db 2023-03-31 18:54:27,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:27,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288927607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:27,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:27,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288927914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:27,941 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/43c115ada62943a1931584c46efb9d86 2023-03-31 18:54:27,953 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/6d7b7f83cc58484e857f5f5a84f0f91a 2023-03-31 18:54:27,957 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/c7cbcabb2f814ae2b21e67660493f0db as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/c7cbcabb2f814ae2b21e67660493f0db 2023-03-31 18:54:27,961 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/c7cbcabb2f814ae2b21e67660493f0db, entries=150, sequenceid=235, filesize=25.3 K 2023-03-31 18:54:27,962 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/43c115ada62943a1931584c46efb9d86 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/43c115ada62943a1931584c46efb9d86 2023-03-31 18:54:27,966 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/43c115ada62943a1931584c46efb9d86, entries=150, sequenceid=235, filesize=11.8 K 2023-03-31 18:54:27,967 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/6d7b7f83cc58484e857f5f5a84f0f91a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d7b7f83cc58484e857f5f5a84f0f91a 2023-03-31 18:54:27,978 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d7b7f83cc58484e857f5f5a84f0f91a, entries=150, sequenceid=235, filesize=11.8 K 2023-03-31 18:54:27,979 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~187.85 KB/192360, heapSize ~478.69 KB/490176, currentSize=13.42 KB/13740 for 8fda10a48807d71ff04525ce9d36a129 in 881ms, sequenceid=235, compaction requested=true 2023-03-31 18:54:27,979 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:27,979 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:27,979 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:27,979 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:27,979 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:27,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:27,991 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:27,992 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:27,992 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:28,001 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:28,001 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:28,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:28,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a5fdd6d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:28,004 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:28,004 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:28,005 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:28,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:28,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:28,008 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:28,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:28,011 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:28,011 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:28,011 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:28,011 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:28,011 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:28,013 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:28,013 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:28,013 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:28,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:28,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:28,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:28,015 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:28,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:28,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:28,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:28,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:28,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:28,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:28,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:28,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:28,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:28,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:28,022 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:28,022 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:28,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:28,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:28,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:28,122 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:28,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:28,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:28,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:28,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:28,128 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:28,128 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:28,128 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:28,128 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:28,129 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:28,129 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:28,132 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:28,132 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:28,132 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:28,132 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:28,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:28,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:28,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:28,133 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:28,133 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:28,133 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:28,134 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:28,134 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:28,134 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:28,134 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:28,134 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-31 18:54:28,135 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:28,135 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:28,135 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:28,135 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:28,135 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:28,135 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:28,165 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331b2afbe7f6837461d8b53ba8f436efab4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b2afbe7f6837461d8b53ba8f436efab4 2023-03-31 18:54:28,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:28,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:28,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288868440, max timestamp=1680288868465Num uniques 859; 2023-03-31 18:54:28,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:28,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288868440, max timestamp=1680288868465Num uniques 859; 2023-03-31 18:54:28,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:28,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288868440, max timestamp=1680288868465Num uniques 859; 2023-03-31 18:54:28,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:28,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680288868501, max timestamp=1680288868514Num uniques 1018; 2023-03-31 18:54:28,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-31 18:54:28,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680288868501, max timestamp=1680288868514Num uniques 1018; 2023-03-31 18:54:28,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680288868501, max timestamp=1680288868514Num uniques 1018; 2023-03-31 18:54:28,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-31 18:54:28,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-31 18:54:28,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680288868547, max timestamp=1680288868571Num uniques 1127; 2023-03-31 18:54:28,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680288868547, max timestamp=1680288868571Num uniques 1127; 2023-03-31 18:54:28,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-31 18:54:28,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-31 18:54:28,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680288868547, max timestamp=1680288868571Num uniques 1127; 2023-03-31 18:54:28,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-31 18:54:28,592 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/18ac124e5d724c52a967e10ed8269aef 2023-03-31 18:54:28,609 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/6f1c8702e5464c6db1f079be34335d50 2023-03-31 18:54:28,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680288868596, max timestamp=1680288868673Num uniques 1136; 2023-03-31 18:54:28,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-03-31 18:54:28,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680288868596, max timestamp=1680288868673Num uniques 1136; 2023-03-31 18:54:28,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-03-31 18:54:28,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680288868596, max timestamp=1680288868673Num uniques 1136; 2023-03-31 18:54:28,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-03-31 18:54:28,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680288868698, max timestamp=1680288868734Num uniques 1195; 2023-03-31 18:54:28,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-03-31 18:54:28,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680288868698, max timestamp=1680288868734Num uniques 1195; 2023-03-31 18:54:28,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-03-31 18:54:28,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680288868698, max timestamp=1680288868734Num uniques 1195; 2023-03-31 18:54:28,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-03-31 18:54:28,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1680288868762, max timestamp=1680288868785Num uniques 1204; 2023-03-31 18:54:28,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=13920, suffix heap size=153184,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=304 2023-03-31 18:54:28,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1680288868762, max timestamp=1680288868785Num uniques 1204; 2023-03-31 18:54:28,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1680288868762, max timestamp=1680288868785Num uniques 1204; 2023-03-31 18:54:28,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=13920, suffix heap size=153184,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=304 2023-03-31 18:54:28,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=13920, suffix heap size=153184,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=304 2023-03-31 18:54:28,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680288868859, max timestamp=1680288868873Num uniques 1163; 2023-03-31 18:54:28,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=7458, suffix heap size=148920,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=163 2023-03-31 18:54:28,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288868859, max timestamp=1680288868873Num uniques 1163; 2023-03-31 18:54:28,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=6870, suffix heap size=148920,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=150 2023-03-31 18:54:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680288868859, max timestamp=1680288868873Num uniques 1163; 2023-03-31 18:54:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=7458, suffix heap size=148920,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=163 2023-03-31 18:54:28,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680288868897, max timestamp=1680288868958Num uniques 1022; 2023-03-31 18:54:28,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-31 18:54:28,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680288868897, max timestamp=1680288868958Num uniques 1009; 2023-03-31 18:54:28,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10162, suffix heap size=132904,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=222 2023-03-31 18:54:28,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680288868897, max timestamp=1680288868958Num uniques 1022; 2023-03-31 18:54:28,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-31 18:54:29,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680288869005, max timestamp=1680288869021Num uniques 1081; 2023-03-31 18:54:29,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-31 18:54:29,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680288869005, max timestamp=1680288869021Num uniques 1081; 2023-03-31 18:54:29,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-31 18:54:29,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680288869005, max timestamp=1680288869021Num uniques 1081; 2023-03-31 18:54:29,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-31 18:54:29,033 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/ff5054b33fc94526af5b7ab63dc2bf7c 2023-03-31 18:54:29,038 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/18ac124e5d724c52a967e10ed8269aef as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/18ac124e5d724c52a967e10ed8269aef 2023-03-31 18:54:29,043 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/18ac124e5d724c52a967e10ed8269aef, entries=50, sequenceid=240, filesize=11.6 K 2023-03-31 18:54:29,044 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/6f1c8702e5464c6db1f079be34335d50 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/6f1c8702e5464c6db1f079be34335d50 2023-03-31 18:54:29,050 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/6f1c8702e5464c6db1f079be34335d50, entries=50, sequenceid=240, filesize=7.1 K 2023-03-31 18:54:29,051 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/ff5054b33fc94526af5b7ab63dc2bf7c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/ff5054b33fc94526af5b7ab63dc2bf7c 2023-03-31 18:54:29,055 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/ff5054b33fc94526af5b7ab63dc2bf7c, entries=50, sequenceid=240, filesize=7.1 K 2023-03-31 18:54:29,056 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=80.51 KB/82440 for 8fda10a48807d71ff04525ce9d36a129 in 922ms, sequenceid=240, compaction requested=true 2023-03-31 18:54:29,056 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:29,056 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:29,058 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d40d806[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:29,069 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:29,069 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,071 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:29,071 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,071 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:29,072 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,072 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:29,073 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,073 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:29,075 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:29,075 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:29,075 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:29,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:29,076 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:29,078 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,078 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,078 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:29,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,079 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=93.93 KB heapSize=263.39 KB 2023-03-31 18:54:29,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:29,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:29,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:29,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:29,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:29,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,080 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,084 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:29,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:29,085 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:29,085 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:29,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:29,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288929147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:29,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288929148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,185 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:29,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:29,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:29,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:29,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:29,192 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:29,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:29,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:29,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,194 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:29,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:29,200 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,200 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:29,200 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:29,200 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:29,202 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,203 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dea5486[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:29,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:29,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:29,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:29,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:29,208 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,208 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:29,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:29,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:29,210 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:29,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:29,212 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:29,212 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:29,212 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:29,212 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:29,212 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:29,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,219 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,222 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,222 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,222 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,222 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,222 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,222 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,222 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,222 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,222 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:29,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:29,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:29,223 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:29,223 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:29,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:29,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288929250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:29,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288929256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,323 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:29,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:29,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:29,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:29,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:29,328 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:29,328 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,328 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:29,328 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:29,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,334 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:29,339 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,339 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,339 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:29,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:29,344 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:29,344 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,344 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:29,344 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:29,344 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:29,344 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,344 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:29,345 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:29,345 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:29,347 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:29,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d6f615[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:29,350 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:29,350 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,352 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,352 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:29,352 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:29,352 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:29,354 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:29,354 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,354 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:29,354 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:29,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:29,355 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:29,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:29,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:29,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:29,355 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:29,360 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,363 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,364 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,368 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,368 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:29,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:29,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:29,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,372 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:29,373 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:29,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,373 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,373 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288929458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288929461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,473 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:29,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:29,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:29,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:29,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:29,476 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:29,476 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,476 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:29,476 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:29,476 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,476 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,478 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,478 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,478 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,478 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:29,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:29,480 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,480 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:29,480 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:29,480 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:29,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:29,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:29,480 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:29,482 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,482 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,483 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,483 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3326f937[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:29,484 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:29,484 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,485 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:29,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:29,485 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:29,485 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,485 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:29,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:29,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,486 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:29,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:29,490 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:29,490 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:29,490 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:29,490 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:29,490 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:29,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,492 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:29,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:29,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,497 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,498 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:29,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,498 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:29,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,523 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303312f3f1ef6944d4310aa992e1ee5e4a426 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312f3f1ef6944d4310aa992e1ee5e4a426 2023-03-31 18:54:29,598 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:29,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:29,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:29,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:29,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:29,601 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:29,601 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,601 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:29,601 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:29,602 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,602 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,603 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,603 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,603 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,604 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:29,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:29,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:29,604 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:29,604 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,604 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:29,605 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,605 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:29,605 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:29,605 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:29,606 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,606 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,606 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,606 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188d97bd[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:29,608 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:29,608 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,609 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:29,612 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,613 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:29,613 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:29,613 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:29,613 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:29,613 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:29,613 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:29,613 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:29,614 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,614 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,614 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,618 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,620 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,620 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,620 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,620 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,620 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,620 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:29,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:29,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,620 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,621 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:29,621 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:29,620 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,721 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:29,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:29,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:29,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:29,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:29,725 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:29,725 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,725 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:29,725 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:29,725 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,725 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,726 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,727 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,727 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,727 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:29,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:29,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:29,729 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,729 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:29,729 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:29,729 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:29,732 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,732 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,733 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,733 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:29,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@578cbe4e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:29,736 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:29,736 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,738 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:29,738 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:29,738 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:29,738 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,738 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:29,739 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,739 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:29,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,740 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,740 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:29,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:29,742 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:29,742 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:29,742 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:29,742 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:29,742 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:29,744 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,744 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,744 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,745 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,746 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,750 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,750 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,750 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,750 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,750 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,751 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:29,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:29,751 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,751 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,751 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,751 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,751 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:29,751 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:29,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:29,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288929768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:29,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288929768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,851 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:29,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:29,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:29,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:29,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:29,855 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:29,855 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,855 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:29,855 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:29,856 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,856 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,859 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,860 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,860 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:29,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:29,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:29,861 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,861 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:29,861 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:29,861 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,862 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:29,862 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:29,863 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:29,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@172a8cf4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:29,865 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:29,865 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,867 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:29,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:29,868 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:29,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:29,870 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:29,870 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,870 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,870 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:29,872 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:29,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:29,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:29,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:29,872 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:29,874 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,874 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,874 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,875 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,876 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,878 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,878 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:29,879 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,879 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:29,878 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:29,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:29,879 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,879 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:29,879 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:29,879 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,879 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,879 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,879 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:29,931 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=408, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/a0683241e09f4b97b7ef9b2226365481 2023-03-31 18:54:29,979 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:29,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:29,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:29,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:29,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:29,983 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:29,983 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,983 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:29,983 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:29,984 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,984 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,986 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:29,986 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,986 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,986 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:29,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:29,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:29,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:29,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:29,988 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:29,988 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:29,988 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:29,988 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,988 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:29,988 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:29,988 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:29,988 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:29,988 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:29,988 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,988 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:29,988 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:29,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,991 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,991 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,991 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:29,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:29,994 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:29,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c13067[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:29,994 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,996 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:29,996 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,996 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:29,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,999 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:29,999 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:29,999 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:30,001 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:30,001 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:30,001 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:30,001 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:30,001 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:30,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,004 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,006 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,008 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,008 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,008 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:30,008 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:30,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:30,009 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,009 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,009 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,009 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,009 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,009 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,009 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:30,009 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:30,109 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:30,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:30,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:30,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:30,112 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:30,113 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,113 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:30,113 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:30,113 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,113 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,114 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,114 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,115 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,115 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:30,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:30,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:30,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:30,117 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,117 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:30,117 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:30,117 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:30,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:30,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67856ddd[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:30,124 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:30,124 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,127 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,127 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:30,127 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:30,127 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:30,128 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:30,128 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,128 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:30,132 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,133 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:30,133 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:30,133 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:30,133 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:30,133 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:30,133 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:30,133 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:30,135 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,135 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,135 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,136 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,138 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,141 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,141 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,142 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:30,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:30,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,142 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:30,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:30,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,142 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:30,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,242 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:30,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:30,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:30,246 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:30,246 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,246 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:30,246 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:30,246 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,246 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,249 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,249 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,249 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,249 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:30,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:30,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:30,251 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,251 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:30,251 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:30,251 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:30,252 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,252 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:30,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c9c1ad[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:30,254 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:30,254 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,258 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,258 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:30,258 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:30,258 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:30,258 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,258 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:30,261 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:30,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:30,263 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:30,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:30,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:30,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:30,263 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:30,264 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,264 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,264 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,266 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,268 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,268 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,268 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,268 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,268 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,268 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,268 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,268 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:30,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:30,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:30,268 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,269 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:30,269 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:30,268 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:30,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288930275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:30,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288930276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/8d270656207741cdba1aa522d85a6a7f 2023-03-31 18:54:30,369 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:30,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:30,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:30,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:30,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:30,372 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:30,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:30,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:30,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,373 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,374 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,374 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,374 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,374 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:30,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:30,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:30,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:30,376 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,376 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:30,376 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:30,376 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:30,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,378 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,379 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,379 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1647bacc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:30,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:30,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,381 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:30,382 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,382 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:30,383 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:30,385 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:30,385 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:30,385 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:30,385 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:30,385 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:30,387 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,387 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,387 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,388 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,389 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:30,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:30,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,393 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,393 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,393 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:30,393 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:30,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:30,493 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:30,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:30,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:30,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:30,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:30,496 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:30,496 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,496 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:30,496 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:30,497 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,497 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,498 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:30,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:30,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:30,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:30,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:30,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:30,502 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,502 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,503 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,503 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:30,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:30,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9ff26b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:30,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,505 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,505 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:30,505 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:30,505 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:30,505 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,506 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:30,508 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:30,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:30,510 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:30,510 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:30,510 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:30,511 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:30,511 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:30,512 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,512 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,512 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,514 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:30,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:30,516 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:30,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,517 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,517 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:30,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,617 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:30,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:30,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:30,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:30,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:30,620 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:30,620 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,620 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:30,620 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:30,620 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,621 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,622 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,622 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,622 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,622 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:30,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:30,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:30,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:30,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:30,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:30,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:30,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:30,625 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,625 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:30,625 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:30,625 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:30,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:30,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:30,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:30,628 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40b88da0[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:30,629 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:30,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,631 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:30,631 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,631 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:30,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,635 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,635 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,635 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:30,638 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:30,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:30,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:30,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:30,638 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:30,640 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,640 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,640 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,641 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,648 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:30,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:30,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,651 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:30,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,651 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:30,652 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:30,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,752 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:30,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:30,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:30,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:30,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:30,755 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:30,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:30,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:30,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,756 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,757 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/70e7204ea83d44c8adaaeb091cde784a 2023-03-31 18:54:30,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:30,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:30,758 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,758 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:30,759 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:30,759 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:30,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,761 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/a0683241e09f4b97b7ef9b2226365481 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a0683241e09f4b97b7ef9b2226365481 2023-03-31 18:54:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5be7e8b3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:30,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:30,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a0683241e09f4b97b7ef9b2226365481, entries=250, sequenceid=408, filesize=39.2 K 2023-03-31 18:54:30,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/8d270656207741cdba1aa522d85a6a7f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8d270656207741cdba1aa522d85a6a7f 2023-03-31 18:54:30,770 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:30,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:30,770 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8d270656207741cdba1aa522d85a6a7f, entries=150, sequenceid=408, filesize=11.9 K 2023-03-31 18:54:30,770 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:30,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:30,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/70e7204ea83d44c8adaaeb091cde784a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/70e7204ea83d44c8adaaeb091cde784a 2023-03-31 18:54:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,772 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:30,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:30,776 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:30,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:30,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:30,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:30,776 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:30,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/70e7204ea83d44c8adaaeb091cde784a, entries=150, sequenceid=408, filesize=11.9 K 2023-03-31 18:54:30,777 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~263.34 KB/269664, currentSize=87.22 KB/89310 for 8fda10a48807d71ff04525ce9d36a129 in 1698ms, sequenceid=408, compaction requested=true 2023-03-31 18:54:30,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:30,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:30,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:30,777 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-31 18:54:30,777 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-31 18:54:30,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:30,778 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:30,778 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:30,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:30,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,782 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:30,782 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:30,782 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:30,782 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:30,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,782 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:30,782 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,782 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,782 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:30,782 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:30,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:30,787 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 226364 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-31 18:54:30,787 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/A is initiating minor compaction (all files) 2023-03-31 18:54:30,787 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/A in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,787 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0dea6dfbb3c848828fdf393e9f89df62, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/1124ed502f064594823b1bfa25c9fda6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0cb5b300305c4ba6b6af87b17761a4f6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ad788f90569242a4a12e6cc7d00da7f5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ee94c2a734f849b49b9f6e1adc375bcd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/070fc876f1604cbb9340607a41add227, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/c7cbcabb2f814ae2b21e67660493f0db, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/18ac124e5d724c52a967e10ed8269aef, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a0683241e09f4b97b7ef9b2226365481] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=221.1 K 2023-03-31 18:54:30,787 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 101348 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-31 18:54:30,788 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/B is initiating minor compaction (all files) 2023-03-31 18:54:30,788 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/B in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,788 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/d5e7da09a84a4811b94d27cd9dbc376b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/9ff667eb30e54c54bbcbb81b12e515d4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/56173d0709d7430384e102a3f2947d92, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/f44c406257924ac6b077a37992146e47, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7fa125b2cf87464591c6ec848e174d46, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/ae35692b5bf44904804b697c6dba7af5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/43c115ada62943a1931584c46efb9d86, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/6f1c8702e5464c6db1f079be34335d50, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8d270656207741cdba1aa522d85a6a7f] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=99.0 K 2023-03-31 18:54:30,789 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 0dea6dfbb3c848828fdf393e9f89df62, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680288859852 2023-03-31 18:54:30,789 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d5e7da09a84a4811b94d27cd9dbc376b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680288859852 2023-03-31 18:54:30,790 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9ff667eb30e54c54bbcbb81b12e515d4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680288859960 2023-03-31 18:54:30,796 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 1124ed502f064594823b1bfa25c9fda6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680288859960 2023-03-31 18:54:30,796 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 56173d0709d7430384e102a3f2947d92, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680288862088 2023-03-31 18:54:30,797 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 0cb5b300305c4ba6b6af87b17761a4f6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680288862088 2023-03-31 18:54:30,797 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f44c406257924ac6b077a37992146e47, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680288863276 2023-03-31 18:54:30,797 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting ad788f90569242a4a12e6cc7d00da7f5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680288863276 2023-03-31 18:54:30,797 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 7fa125b2cf87464591c6ec848e174d46, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680288863907 2023-03-31 18:54:30,797 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting ee94c2a734f849b49b9f6e1adc375bcd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680288863907 2023-03-31 18:54:30,798 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ae35692b5bf44904804b697c6dba7af5, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680288866064 2023-03-31 18:54:30,799 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 43c115ada62943a1931584c46efb9d86, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680288866154 2023-03-31 18:54:30,799 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 070fc876f1604cbb9340607a41add227, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680288866064 2023-03-31 18:54:30,799 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 6f1c8702e5464c6db1f079be34335d50, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680288867300 2023-03-31 18:54:30,800 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c7cbcabb2f814ae2b21e67660493f0db, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680288866154 2023-03-31 18:54:30,800 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 8d270656207741cdba1aa522d85a6a7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288869041 2023-03-31 18:54:30,800 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 18ac124e5d724c52a967e10ed8269aef, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680288867300 2023-03-31 18:54:30,801 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a0683241e09f4b97b7ef9b2226365481, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288869041 2023-03-31 18:54:30,834 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#B#compaction#148 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:30,840 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#A#compaction#147 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:30,882 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:30,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:30,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:30,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:30,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:30,885 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:30,885 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:30,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:30,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,888 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,888 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:30,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:30,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:30,888 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:30,888 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:30,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:30,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:30,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:30,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:30,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:30,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:30,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:30,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:30,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:30,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:30,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:30,896 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-31 18:54:30,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:30,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:30,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:30,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:30,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:30,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:31,272 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/012337e253ef42398b2e8c5e87d5cb31 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/012337e253ef42398b2e8c5e87d5cb31 2023-03-31 18:54:31,283 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/B of 8fda10a48807d71ff04525ce9d36a129 into 012337e253ef42398b2e8c5e87d5cb31(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:31,283 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:31,283 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/B, priority=7, startTime=1680288870777; duration=0sec 2023-03-31 18:54:31,283 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:31,283 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-03-31 18:54:31,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:31,292 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:31,294 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 110776 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-03-31 18:54:31,295 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/C is initiating minor compaction (all files) 2023-03-31 18:54:31,295 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/C in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:31,295 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/be0494c20f1243e3b57287a35c39c561, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/3a087997f23f4558b444b2eb1c95138d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/a26f40721fed449bb3906b553bee29a1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/0d422e4536754976ab48e09cbf35865d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/d55046dd82b24ace908be96a1cb1544a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e95fcb9d8ad94ea99bfa1fcba6e06ca5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e7dc4248870d4591879d3c29f828af69, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d7b7f83cc58484e857f5f5a84f0f91a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/ff5054b33fc94526af5b7ab63dc2bf7c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/70e7204ea83d44c8adaaeb091cde784a] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=108.2 K 2023-03-31 18:54:31,295 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting be0494c20f1243e3b57287a35c39c561, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680288857657 2023-03-31 18:54:31,296 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3a087997f23f4558b444b2eb1c95138d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680288859852 2023-03-31 18:54:31,296 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a26f40721fed449bb3906b553bee29a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680288859960 2023-03-31 18:54:31,299 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 0d422e4536754976ab48e09cbf35865d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680288862088 2023-03-31 18:54:31,300 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d55046dd82b24ace908be96a1cb1544a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680288863276 2023-03-31 18:54:31,300 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e95fcb9d8ad94ea99bfa1fcba6e06ca5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680288863907 2023-03-31 18:54:31,301 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e7dc4248870d4591879d3c29f828af69, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680288866065 2023-03-31 18:54:31,302 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 6d7b7f83cc58484e857f5f5a84f0f91a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680288866154 2023-03-31 18:54:31,302 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ff5054b33fc94526af5b7ab63dc2bf7c, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680288867300 2023-03-31 18:54:31,303 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 70e7204ea83d44c8adaaeb091cde784a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288869041 2023-03-31 18:54:31,323 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303319b824ff9cb7447e7b1b5c71bf87e5ec9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319b824ff9cb7447e7b1b5c71bf87e5ec9 2023-03-31 18:54:31,324 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#C#compaction#150 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:31,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:31,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288931335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:31,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:31,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288931336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:31,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:31,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288931441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:31,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:31,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288931444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:31,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:31,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288931644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:31,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:31,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288931646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:31,700 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/77ae87e378574405a20e9b717cbd5521 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/77ae87e378574405a20e9b717cbd5521 2023-03-31 18:54:31,706 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/A of 8fda10a48807d71ff04525ce9d36a129 into 77ae87e378574405a20e9b717cbd5521(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:31,706 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:31,706 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/A, priority=7, startTime=1680288870777; duration=0sec 2023-03-31 18:54:31,707 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:31,734 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=424, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/9c1da7fbe96845aca57b1df339a489ab 2023-03-31 18:54:31,755 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/f4748cab75d84dbbb896eca70bbbd22d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/f4748cab75d84dbbb896eca70bbbd22d 2023-03-31 18:54:31,761 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/C of 8fda10a48807d71ff04525ce9d36a129 into f4748cab75d84dbbb896eca70bbbd22d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:31,761 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:31,761 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/C, priority=6, startTime=1680288870777; duration=0sec 2023-03-31 18:54:31,761 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:31,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:31,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:31,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288931949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:31,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288931949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,150 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/7707369c4b524d70820e342e9e00a840 2023-03-31 18:54:32,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288932451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:32,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288932454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,565 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/aad72891c85e4aa2ad6a648db10c4717 2023-03-31 18:54:32,570 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/9c1da7fbe96845aca57b1df339a489ab as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9c1da7fbe96845aca57b1df339a489ab 2023-03-31 18:54:32,575 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9c1da7fbe96845aca57b1df339a489ab, entries=150, sequenceid=424, filesize=25.4 K 2023-03-31 18:54:32,576 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/7707369c4b524d70820e342e9e00a840 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7707369c4b524d70820e342e9e00a840 2023-03-31 18:54:32,580 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7707369c4b524d70820e342e9e00a840, entries=150, sequenceid=424, filesize=11.9 K 2023-03-31 18:54:32,581 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/aad72891c85e4aa2ad6a648db10c4717 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/aad72891c85e4aa2ad6a648db10c4717 2023-03-31 18:54:32,586 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/aad72891c85e4aa2ad6a648db10c4717, entries=150, sequenceid=424, filesize=11.9 K 2023-03-31 18:54:32,587 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 8fda10a48807d71ff04525ce9d36a129 in 1691ms, sequenceid=424, compaction requested=false 2023-03-31 18:54:32,587 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:32,587 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:32,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,588 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:32,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:32,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705db8d9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:32,590 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:32,590 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:32,591 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:32,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:32,592 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,592 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:32,593 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,593 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:32,593 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:32,593 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:32,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:32,595 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:32,595 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:32,595 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:32,595 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:32,595 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:32,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:32,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:32,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:32,598 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:32,599 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,601 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,601 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:32,601 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:32,601 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:32,601 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:32,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:32,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:32,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:32,601 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:32,601 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,602 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:32,602 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:32,602 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:32,602 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:32,602 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:32,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:32,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:32,702 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:32,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:32,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:32,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:32,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:32,705 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:32,705 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:32,705 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:32,705 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:32,705 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:32,705 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:32,706 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:32,706 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:32,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:32,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:32,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:32,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:32,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:32,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:32,708 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:32,708 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:32,708 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:32,709 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-31 18:54:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:33,126 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331c36fa9dd9d10407b8774c427823afead to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c36fa9dd9d10407b8774c427823afead 2023-03-31 18:54:33,131 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=444, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/7889825a6e664eed8a87bedc5938be6b 2023-03-31 18:54:33,154 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/8369abd9276a4388bf859ee16c1e29ea 2023-03-31 18:54:33,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:33,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:33,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:33,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288933480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:33,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288933484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,565 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/2b70f532d2cc4a5a82acc4d28ba72c54 2023-03-31 18:54:33,570 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/7889825a6e664eed8a87bedc5938be6b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/7889825a6e664eed8a87bedc5938be6b 2023-03-31 18:54:33,574 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/7889825a6e664eed8a87bedc5938be6b, entries=150, sequenceid=444, filesize=25.4 K 2023-03-31 18:54:33,575 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/8369abd9276a4388bf859ee16c1e29ea as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8369abd9276a4388bf859ee16c1e29ea 2023-03-31 18:54:33,578 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8369abd9276a4388bf859ee16c1e29ea, entries=150, sequenceid=444, filesize=11.9 K 2023-03-31 18:54:33,581 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/2b70f532d2cc4a5a82acc4d28ba72c54 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2b70f532d2cc4a5a82acc4d28ba72c54 2023-03-31 18:54:33,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:33,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288933585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,585 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2b70f532d2cc4a5a82acc4d28ba72c54, entries=150, sequenceid=444, filesize=11.9 K 2023-03-31 18:54:33,586 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8fda10a48807d71ff04525ce9d36a129 in 877ms, sequenceid=444, compaction requested=true 2023-03-31 18:54:33,586 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:33,586 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:33,586 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:33,586 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:33,586 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,586 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:33,586 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:33,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-31 18:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,588 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:33,588 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:33,588 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:33,588 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:33,588 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:33,588 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:33,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3220013[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:33,590 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:33,590 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,592 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,592 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,592 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:33,594 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:33,597 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:33,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:33,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:33,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:33,597 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:33,600 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,600 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,600 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,601 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,602 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,607 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,608 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:33,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:33,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:33,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:33,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:33,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:33,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:33,608 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:33,608 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:33,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:33,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:33,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288933625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,708 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:33,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:33,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:33,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:33,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:33,721 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:33,721 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:33,721 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:33,721 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:33,722 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,722 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,723 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,723 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:33,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:33,723 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,723 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:33,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:33,724 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:33,724 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:33,725 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:33,725 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:33,725 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:33,725 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:33,725 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,725 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:33,725 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:33,728 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,728 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,728 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,728 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288933727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773485a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:33,730 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:33,730 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,731 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,731 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,731 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:33,732 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:33,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:33,734 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:33,734 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:33,734 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:33,734 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:33,734 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:33,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:33,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:33,736 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,737 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,741 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:33,741 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:33,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:33,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:33,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:33,741 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:33,741 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,741 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,741 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,741 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:33,741 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,741 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:33,741 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,741 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,741 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:33,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:33,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288933786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,842 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:33,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:33,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:33,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:33,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:33,845 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:33,845 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:33,845 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:33,845 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:33,845 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,845 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,846 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,846 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:33,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:33,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:33,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:33,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:33,848 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:33,848 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:33,848 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:33,848 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:33,850 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:33,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63359a77[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:33,851 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:33,851 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,852 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,852 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:33,852 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,852 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:33,852 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:33,852 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:33,855 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:33,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:33,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:33,857 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:33,858 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:33,858 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:33,858 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:33,858 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:33,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:33,859 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:33,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,860 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,862 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,862 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,862 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,862 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,862 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,862 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:33,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:33,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:33,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:33,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:33,862 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,862 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:33,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:33,863 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:33,863 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:33,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:33,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:33,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288933930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,963 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:33,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:33,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:33,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:33,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:33,968 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:33,968 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:33,968 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:33,968 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:33,968 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,968 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,969 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,970 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:33,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:33,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:33,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:33,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:33,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:33,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:33,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:33,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:33,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:33,971 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:33,972 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:33,972 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:33,972 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:33,972 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:33,972 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:33,972 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,972 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:33,972 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:33,974 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,974 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,974 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,974 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:33,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb36db8[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:33,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:33,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,977 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:33,978 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,978 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:33,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,979 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,979 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:33,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:33,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:33,983 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:33,983 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:33,983 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:33,983 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:33,983 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:33,985 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,985 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:33,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,985 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:33,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:33,986 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,988 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,996 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:33,996 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,996 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,996 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:33,996 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:33,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:33,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:33,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:33,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:33,996 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:33,997 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:33,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:33,997 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,997 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:33,997 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:33,997 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:33,997 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:33,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:33,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,005 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331f1ce28daee624f528f83d4ddd5078de6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f1ce28daee624f528f83d4ddd5078de6 2023-03-31 18:54:34,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:34,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288934087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,097 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:34,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:34,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:34,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:34,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:34,101 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:34,101 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,101 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:34,101 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:34,101 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,101 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,102 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,103 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,103 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,103 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:34,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:34,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:34,104 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:34,104 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,104 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:34,104 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:34,105 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,105 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:34,105 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:34,105 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:34,106 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,107 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,107 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d02c8e5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:34,108 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:34,108 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,109 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:34,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:34,110 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:34,110 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,110 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:34,111 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,112 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,112 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,112 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:34,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:34,114 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:34,114 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:34,114 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:34,114 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:34,114 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:34,116 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,116 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,116 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,116 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,117 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,119 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:34,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:34,119 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,120 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:34,120 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:34,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:34,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:34,220 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:34,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:34,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:34,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:34,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:34,223 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:34,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:34,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:34,224 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,224 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,225 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:34,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:34,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:34,227 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,227 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:34,227 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:34,227 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:34,230 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ae6c87[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:34,232 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:34,232 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,233 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:34,233 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,233 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:34,234 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:34,234 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,234 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,234 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288934233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:34,237 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:34,237 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:34,237 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:34,237 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:34,237 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:34,238 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,238 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,238 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,239 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,240 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,243 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,243 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,243 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,243 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,243 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:34,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:34,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:34,243 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,243 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,244 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,244 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,244 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,244 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:34,244 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:34,344 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:34,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:34,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:34,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:34,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:34,348 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:34,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:34,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:34,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,350 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,350 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,351 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,351 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:34,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:34,360 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,360 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:34,360 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:34,360 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:34,361 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:34,361 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,361 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:34,361 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:34,362 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,363 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,363 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:34,364 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f07b6c7[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:34,364 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,365 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:34,365 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:34,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:34,366 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,366 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:34,367 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:34,367 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,367 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,367 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:34,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:34,370 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:34,370 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:34,370 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:34,370 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:34,370 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:34,371 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,371 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,371 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,373 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,375 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:34,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:34,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,376 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,376 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:34,376 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:34,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,411 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=460, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/67f7cc09a34e4daca5902cf0e5861c19 2023-03-31 18:54:34,476 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:34,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:34,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:34,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:34,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:34,479 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:34,479 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,479 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:34,479 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:34,480 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,480 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,485 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,485 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,485 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,485 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:34,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:34,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:34,486 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,486 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:34,486 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:34,486 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:34,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:34,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:34,487 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:34,488 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:34,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5abcd232[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:34,490 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:34,490 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,491 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,491 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:34,491 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,491 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:34,491 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:34,491 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:34,492 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,492 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:34,492 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,492 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:34,494 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:34,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:34,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:34,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:34,494 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:34,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,497 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,501 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:34,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:34,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,501 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:34,501 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:34,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,501 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:34,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288934592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,601 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:34,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:34,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:34,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:34,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:34,607 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:34,607 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,607 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:34,607 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:34,608 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,608 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,609 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,609 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:34,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:34,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:34,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:34,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:34,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:34,612 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,612 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:34,612 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:34,612 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:34,614 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,615 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,616 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,616 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:34,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21826c60[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:34,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:34,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,619 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:34,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:34,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:34,620 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,620 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:34,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:34,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:34,624 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:34,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:34,625 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:34,625 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:34,625 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:34,626 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,626 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,626 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,629 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,631 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,634 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:34,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:34,634 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,635 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:34,635 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:34,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:34,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58418 deadline: 1680288934679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,683 DEBUG [Thread-869] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18244 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:34,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:34,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58424 deadline: 1680288934691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,695 DEBUG [Thread-865] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18251 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:34,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:34,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58412 deadline: 1680288934700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,704 DEBUG [Thread-861] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18254 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:34,735 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:34,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:34,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:34,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:34,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:34,739 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:34,739 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,739 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:34,739 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:34,739 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,739 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288934739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,742 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,742 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,742 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,742 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:34,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:34,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:34,743 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,744 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:34,744 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:34,744 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:34,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:34,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:34,744 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:34,745 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,746 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,747 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,747 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:34,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:34,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22cbe683[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:34,747 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:34,747 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,748 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:34,749 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,749 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:34,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:34,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,750 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,750 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:34,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:34,752 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:34,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:34,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:34,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:34,753 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:34,755 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,755 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,755 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:34,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:34,755 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,759 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,759 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:34,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:34,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:34,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,760 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:34,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:34,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:34,760 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:34,761 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:34,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/543cb013bd9941dd9b1eb8bd35ef1243 2023-03-31 18:54:34,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/df4e8762cd03444493ef83562ae68350 2023-03-31 18:54:34,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/67f7cc09a34e4daca5902cf0e5861c19 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/67f7cc09a34e4daca5902cf0e5861c19 2023-03-31 18:54:34,857 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/67f7cc09a34e4daca5902cf0e5861c19, entries=200, sequenceid=460, filesize=32.3 K 2023-03-31 18:54:34,857 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/543cb013bd9941dd9b1eb8bd35ef1243 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/543cb013bd9941dd9b1eb8bd35ef1243 2023-03-31 18:54:34,861 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:34,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/543cb013bd9941dd9b1eb8bd35ef1243, entries=150, sequenceid=460, filesize=11.9 K 2023-03-31 18:54:34,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/df4e8762cd03444493ef83562ae68350 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/df4e8762cd03444493ef83562ae68350 2023-03-31 18:54:34,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:34,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:34,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:34,866 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:34,866 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,866 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:34,866 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:34,866 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,866 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,867 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,867 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:34,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:34,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:34,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:34,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/df4e8762cd03444493ef83562ae68350, entries=150, sequenceid=460, filesize=11.9 K 2023-03-31 18:54:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:34,870 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8fda10a48807d71ff04525ce9d36a129 in 1282ms, sequenceid=460, compaction requested=true 2023-03-31 18:54:34,870 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:34,870 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:34,871 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:54:34,871 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:54:34,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:34,871 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:34,871 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:34,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:34,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:34,872 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:34,872 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:34,872 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:34,873 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111736 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:54:34,873 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:54:34,873 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/B is initiating minor compaction (all files) 2023-03-31 18:54:34,873 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/B in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,874 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/012337e253ef42398b2e8c5e87d5cb31, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7707369c4b524d70820e342e9e00a840, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8369abd9276a4388bf859ee16c1e29ea, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/543cb013bd9941dd9b1eb8bd35ef1243] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=48.1 K 2023-03-31 18:54:34,873 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/A is initiating minor compaction (all files) 2023-03-31 18:54:34,874 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/A in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,874 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/77ae87e378574405a20e9b717cbd5521, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9c1da7fbe96845aca57b1df339a489ab, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/7889825a6e664eed8a87bedc5938be6b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/67f7cc09a34e4daca5902cf0e5861c19] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=109.1 K 2023-03-31 18:54:34,874 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 012337e253ef42398b2e8c5e87d5cb31, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288869041 2023-03-31 18:54:34,875 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 7707369c4b524d70820e342e9e00a840, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680288869140 2023-03-31 18:54:34,875 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 77ae87e378574405a20e9b717cbd5521, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288869041 2023-03-31 18:54:34,875 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 8369abd9276a4388bf859ee16c1e29ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680288871326 2023-03-31 18:54:34,875 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9c1da7fbe96845aca57b1df339a489ab, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680288869140 2023-03-31 18:54:34,876 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 543cb013bd9941dd9b1eb8bd35ef1243, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1680288873461 2023-03-31 18:54:34,876 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:34,876 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 7889825a6e664eed8a87bedc5938be6b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680288871326 2023-03-31 18:54:34,876 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:34,876 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,876 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:34,876 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-31 18:54:34,876 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:34,877 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 67f7cc09a34e4daca5902cf0e5861c19, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1680288873461 2023-03-31 18:54:34,877 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:34,877 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:34,877 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:34,877 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:34,877 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:34,889 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#B#compaction#159 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:34,901 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#A#compaction#161 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:34,916 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033160a581c88a2c4210aec76921c6326a96 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033160a581c88a2c4210aec76921c6326a96 2023-03-31 18:54:34,920 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/2fdaa3a5489a420da60649e4ae0c0e9b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/2fdaa3a5489a420da60649e4ae0c0e9b 2023-03-31 18:54:34,926 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/B of 8fda10a48807d71ff04525ce9d36a129 into 2fdaa3a5489a420da60649e4ae0c0e9b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:34,926 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:34,926 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/B, priority=12, startTime=1680288874871; duration=0sec 2023-03-31 18:54:34,926 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:34,926 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:54:34,928 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:54:34,928 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/C is initiating minor compaction (all files) 2023-03-31 18:54:34,928 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/C in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:34,928 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/f4748cab75d84dbbb896eca70bbbd22d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/aad72891c85e4aa2ad6a648db10c4717, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2b70f532d2cc4a5a82acc4d28ba72c54, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/df4e8762cd03444493ef83562ae68350] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=48.1 K 2023-03-31 18:54:34,928 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f4748cab75d84dbbb896eca70bbbd22d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288869041 2023-03-31 18:54:34,929 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting aad72891c85e4aa2ad6a648db10c4717, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680288869140 2023-03-31 18:54:34,929 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2b70f532d2cc4a5a82acc4d28ba72c54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680288871326 2023-03-31 18:54:34,930 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting df4e8762cd03444493ef83562ae68350, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1680288873461 2023-03-31 18:54:34,954 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#C#compaction#162 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:35,324 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=476, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/723b130b5ad74236a94bf6cfc77ba376 2023-03-31 18:54:35,332 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/41952185611f4769becf869a7045da20 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/41952185611f4769becf869a7045da20 2023-03-31 18:54:35,343 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/A of 8fda10a48807d71ff04525ce9d36a129 into 41952185611f4769becf869a7045da20(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:35,343 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:35,343 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/A, priority=12, startTime=1680288874870; duration=0sec 2023-03-31 18:54:35,343 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:35,366 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/eececc4a8e9a4bebbe2d7f42f57d6b58 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/eececc4a8e9a4bebbe2d7f42f57d6b58 2023-03-31 18:54:35,372 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/C of 8fda10a48807d71ff04525ce9d36a129 into eececc4a8e9a4bebbe2d7f42f57d6b58(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:35,372 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:35,373 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/C, priority=12, startTime=1680288874871; duration=0sec 2023-03-31 18:54:35,373 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:35,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:35,606 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:35,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:35,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288935633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,752 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/5b1e4e590b844884891081c902b1c35b 2023-03-31 18:54:35,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:35,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288935735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:35,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58396 deadline: 1680288935752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,767 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/b8ff6d58e8164c1e820e9340e2d71d4e 2023-03-31 18:54:35,772 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/723b130b5ad74236a94bf6cfc77ba376 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/723b130b5ad74236a94bf6cfc77ba376 2023-03-31 18:54:35,776 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/723b130b5ad74236a94bf6cfc77ba376, entries=150, sequenceid=476, filesize=25.4 K 2023-03-31 18:54:35,777 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/5b1e4e590b844884891081c902b1c35b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/5b1e4e590b844884891081c902b1c35b 2023-03-31 18:54:35,783 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/5b1e4e590b844884891081c902b1c35b, entries=150, sequenceid=476, filesize=11.9 K 2023-03-31 18:54:35,784 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/b8ff6d58e8164c1e820e9340e2d71d4e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/b8ff6d58e8164c1e820e9340e2d71d4e 2023-03-31 18:54:35,788 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/b8ff6d58e8164c1e820e9340e2d71d4e, entries=150, sequenceid=476, filesize=11.9 K 2023-03-31 18:54:35,789 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8fda10a48807d71ff04525ce9d36a129 in 913ms, sequenceid=476, compaction requested=false 2023-03-31 18:54:35,789 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:35,789 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:35,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:35,791 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:35,791 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:35,791 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:35,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:35,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f8dd93[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:35,793 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:35,793 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:35,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:35,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:35,797 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:35,797 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:35,797 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:35,798 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,798 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:35,799 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:35,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:35,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:35,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:35,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:35,802 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:35,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:35,803 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:35,803 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:35,803 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:35,804 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:35,804 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:35,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:35,804 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:35,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:35,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,807 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:35,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:35,810 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:35,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:35,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:35,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:35,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:35,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:35,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:35,810 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:35,811 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:35,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:35,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:35,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:35,911 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:35,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:35,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:35,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:35,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:35,914 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:35,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:35,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:35,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:35,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:35,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:35,916 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:35,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:35,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:35,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:35,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:35,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:35,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:35,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:35,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:35,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:35,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:35,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:35,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:35,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:35,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:35,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:35,918 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:35,918 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. started... 2023-03-31 18:54:35,918 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-31 18:54:35,918 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:35,918 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:35,919 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:35,919 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:35,919 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:35,919 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:35,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:35,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. as already flushing 2023-03-31 18:54:35,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:35,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288935985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:36,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:36,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288936089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:36,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:36,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288936292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:36,339 DEBUG [Thread-874] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:63858 2023-03-31 18:54:36,339 DEBUG [Thread-874] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:36,339 DEBUG [Thread-872] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:63858 2023-03-31 18:54:36,339 DEBUG [Thread-872] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:36,339 DEBUG [Thread-876] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:63858 2023-03-31 18:54:36,339 DEBUG [Thread-876] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:36,342 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331a91796280d694db692b3e67e6a0c7047 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a91796280d694db692b3e67e6a0c7047 2023-03-31 18:54:36,345 DEBUG [Thread-878] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2efdb02f to 127.0.0.1:63858 2023-03-31 18:54:36,345 DEBUG [Thread-878] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:36,347 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=495, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/25eac5d5f3ac4144b859a3a2d5ffceab 2023-03-31 18:54:36,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:36,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288936594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:36,757 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/e3a036293b8845d19fc7d19bcb262ef7 2023-03-31 18:54:37,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:37,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58384 deadline: 1680288937095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,170 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/9c7af872a65f46378b092e702d00b8c5 2023-03-31 18:54:37,174 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/25eac5d5f3ac4144b859a3a2d5ffceab as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/25eac5d5f3ac4144b859a3a2d5ffceab 2023-03-31 18:54:37,178 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/25eac5d5f3ac4144b859a3a2d5ffceab, entries=150, sequenceid=495, filesize=25.4 K 2023-03-31 18:54:37,179 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/e3a036293b8845d19fc7d19bcb262ef7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3a036293b8845d19fc7d19bcb262ef7 2023-03-31 18:54:37,183 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3a036293b8845d19fc7d19bcb262ef7, entries=150, sequenceid=495, filesize=11.9 K 2023-03-31 18:54:37,183 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/9c7af872a65f46378b092e702d00b8c5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/9c7af872a65f46378b092e702d00b8c5 2023-03-31 18:54:37,187 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/9c7af872a65f46378b092e702d00b8c5, entries=150, sequenceid=495, filesize=11.9 K 2023-03-31 18:54:37,188 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8fda10a48807d71ff04525ce9d36a129 in 1270ms, sequenceid=495, compaction requested=true 2023-03-31 18:54:37,188 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:37,188 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:37,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:37,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:37,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:37,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:37,189 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:37,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:37,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce9a6fc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:37,191 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:37,191 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:37,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:37,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:37,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:37,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:37,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:37,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:37,195 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:37,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:37,197 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:37,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:37,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:37,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:37,197 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:37,199 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:37,199 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:37,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:37,199 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:37,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:37,199 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:37,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:37,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:37,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:37,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:37,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:37,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:37,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:37,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:37,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:37,202 DEBUG [Thread-871] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:37,202 DEBUG [Thread-871] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:37,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:37,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:37,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:37,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:37,302 DEBUG [Thread-871] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:37,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:37,717 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-31 18:54:37,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:37,757 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-31 18:54:37,757 DEBUG [Thread-863] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:63858 2023-03-31 18:54:37,757 DEBUG [Thread-863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:37,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:37,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:37,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:37,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:37,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:37,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:38,098 DEBUG [Thread-867] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:63858 2023-03-31 18:54:38,098 DEBUG [Thread-867] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:38,170 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303311078e8f68777469eb89c02ba78fadd64 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311078e8f68777469eb89c02ba78fadd64 2023-03-31 18:54:38,575 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=512, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/9a5df6fad66d439a9ef881456157ff84 2023-03-31 18:54:38,987 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/e3b9694b8ac74f3fade66e7327540eeb 2023-03-31 18:54:39,397 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/7e1e23b4b793436f8e555a026a017768 2023-03-31 18:54:39,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/9a5df6fad66d439a9ef881456157ff84 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9a5df6fad66d439a9ef881456157ff84 2023-03-31 18:54:39,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9a5df6fad66d439a9ef881456157ff84, entries=150, sequenceid=512, filesize=25.4 K 2023-03-31 18:54:39,404 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/e3b9694b8ac74f3fade66e7327540eeb as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3b9694b8ac74f3fade66e7327540eeb 2023-03-31 18:54:39,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3b9694b8ac74f3fade66e7327540eeb, entries=150, sequenceid=512, filesize=11.9 K 2023-03-31 18:54:39,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/7e1e23b4b793436f8e555a026a017768 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/7e1e23b4b793436f8e555a026a017768 2023-03-31 18:54:39,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/7e1e23b4b793436f8e555a026a017768, entries=150, sequenceid=512, filesize=11.9 K 2023-03-31 18:54:39,413 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=6.71 KB/6870 for 8fda10a48807d71ff04525ce9d36a129 in 1656ms, sequenceid=512, compaction requested=true 2023-03-31 18:54:39,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:39,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:39,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:39,413 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:54:39,413 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:54:39,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:39,414 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:54:39,414 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:54:39,414 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/A is initiating minor compaction (all files) 2023-03-31 18:54:39,414 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/B is initiating minor compaction (all files) 2023-03-31 18:54:39,414 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/A in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:39,414 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/B in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:39,415 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/41952185611f4769becf869a7045da20, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/723b130b5ad74236a94bf6cfc77ba376, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/25eac5d5f3ac4144b859a3a2d5ffceab, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9a5df6fad66d439a9ef881456157ff84] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=102.4 K 2023-03-31 18:54:39,415 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/2fdaa3a5489a420da60649e4ae0c0e9b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/5b1e4e590b844884891081c902b1c35b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3a036293b8845d19fc7d19bcb262ef7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3b9694b8ac74f3fade66e7327540eeb] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=48.2 K 2023-03-31 18:54:39,415 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 41952185611f4769becf869a7045da20, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1680288873461 2023-03-31 18:54:39,415 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2fdaa3a5489a420da60649e4ae0c0e9b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1680288873461 2023-03-31 18:54:39,415 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 723b130b5ad74236a94bf6cfc77ba376, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680288873601 2023-03-31 18:54:39,415 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 5b1e4e590b844884891081c902b1c35b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680288873601 2023-03-31 18:54:39,416 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 25eac5d5f3ac4144b859a3a2d5ffceab, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680288875626 2023-03-31 18:54:39,416 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e3a036293b8845d19fc7d19bcb262ef7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680288875626 2023-03-31 18:54:39,416 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9a5df6fad66d439a9ef881456157ff84, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680288875979 2023-03-31 18:54:39,416 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e3b9694b8ac74f3fade66e7327540eeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680288875979 2023-03-31 18:54:39,438 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#B#compaction#171 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:39,444 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#A#compaction#172 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:39,450 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/78cb9670786449c1ac3485b185f665dd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/78cb9670786449c1ac3485b185f665dd 2023-03-31 18:54:39,456 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/B of 8fda10a48807d71ff04525ce9d36a129 into 78cb9670786449c1ac3485b185f665dd(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:39,456 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:39,456 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/B, priority=12, startTime=1680288879413; duration=0sec 2023-03-31 18:54:39,456 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:39,456 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:54:39,457 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:54:39,457 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 8fda10a48807d71ff04525ce9d36a129/C is initiating minor compaction (all files) 2023-03-31 18:54:39,457 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8fda10a48807d71ff04525ce9d36a129/C in TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:39,457 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/eececc4a8e9a4bebbe2d7f42f57d6b58, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/b8ff6d58e8164c1e820e9340e2d71d4e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/9c7af872a65f46378b092e702d00b8c5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/7e1e23b4b793436f8e555a026a017768] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp, totalSize=48.2 K 2023-03-31 18:54:39,458 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting eececc4a8e9a4bebbe2d7f42f57d6b58, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1680288873461 2023-03-31 18:54:39,458 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b8ff6d58e8164c1e820e9340e2d71d4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680288873601 2023-03-31 18:54:39,458 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9c7af872a65f46378b092e702d00b8c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680288875626 2023-03-31 18:54:39,459 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 7e1e23b4b793436f8e555a026a017768, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680288875979 2023-03-31 18:54:39,467 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 8fda10a48807d71ff04525ce9d36a129#C#compaction#173 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:39,875 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/bfee61f266ba497187e65d301256db40 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/bfee61f266ba497187e65d301256db40 2023-03-31 18:54:39,879 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/C of 8fda10a48807d71ff04525ce9d36a129 into bfee61f266ba497187e65d301256db40(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:39,879 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:39,879 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/C, priority=12, startTime=1680288879413; duration=0sec 2023-03-31 18:54:39,879 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:40,266 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/dac6ed4f3ee346088712d41073355ff3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/dac6ed4f3ee346088712d41073355ff3 2023-03-31 18:54:40,270 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8fda10a48807d71ff04525ce9d36a129/A of 8fda10a48807d71ff04525ce9d36a129 into dac6ed4f3ee346088712d41073355ff3(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:40,270 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:40,270 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129., storeName=8fda10a48807d71ff04525ce9d36a129/A, priority=12, startTime=1680288879413; duration=0sec 2023-03-31 18:54:40,270 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:44,702 DEBUG [Thread-869] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:63858 2023-03-31 18:54:44,702 DEBUG [Thread-869] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:44,756 DEBUG [Thread-865] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:63858 2023-03-31 18:54:44,756 DEBUG [Thread-865] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:44,790 DEBUG [Thread-861] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21281a37 to 127.0.0.1:63858 2023-03-31 18:54:44,790 DEBUG [Thread-861] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 13 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 198 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 8 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 4587 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 4508 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1932 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5796 rows 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1915 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5739 rows 2023-03-31 18:54:44,790 INFO [Listener at localhost/43327] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 18:54:44,790 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7829b43e to 127.0.0.1:63858 2023-03-31 18:54:44,791 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:54:44,792 INFO [Listener at localhost/43327] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 18:54:44,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 18:54:44,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:44,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 18:54:44,797 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288884797"}]},"ts":"1680288884797"} 2023-03-31 18:54:44,798 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 18:54:44,799 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 18:54:44,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, UNASSIGN}] 2023-03-31 18:54:44,801 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, UNASSIGN 2023-03-31 18:54:44,802 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=8fda10a48807d71ff04525ce9d36a129, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:44,802 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288884802"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288884802"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288884802"}]},"ts":"1680288884802"} 2023-03-31 18:54:44,803 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:54:44,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 18:54:44,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:44,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8fda10a48807d71ff04525ce9d36a129, disabling compactions & flushes 2023-03-31 18:54:44,955 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:44,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:44,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. after waiting 0 ms 2023-03-31 18:54:44,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:44,955 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8fda10a48807d71ff04525ce9d36a129 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 18:54:44,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=A 2023-03-31 18:54:44,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:44,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=B 2023-03-31 18:54:44,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:44,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8fda10a48807d71ff04525ce9d36a129, store=C 2023-03-31 18:54:44,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:45,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 18:54:45,368 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303315aa2227ec4fa492888b5132bdb476498 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315aa2227ec4fa492888b5132bdb476498 2023-03-31 18:54:45,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 18:54:45,774 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=522, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/2801395e237c4ce3bd60b2db77964cc2 2023-03-31 18:54:45,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 18:54:46,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/3f8e0c34cb14459a9c888855ca28bd07 2023-03-31 18:54:46,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/762d1b01956f4dcab60292bc8066b26a 2023-03-31 18:54:46,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/A/2801395e237c4ce3bd60b2db77964cc2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/2801395e237c4ce3bd60b2db77964cc2 2023-03-31 18:54:46,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/2801395e237c4ce3bd60b2db77964cc2, entries=100, sequenceid=522, filesize=18.6 K 2023-03-31 18:54:46,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/B/3f8e0c34cb14459a9c888855ca28bd07 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/3f8e0c34cb14459a9c888855ca28bd07 2023-03-31 18:54:46,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/3f8e0c34cb14459a9c888855ca28bd07, entries=100, sequenceid=522, filesize=9.5 K 2023-03-31 18:54:46,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/.tmp/C/762d1b01956f4dcab60292bc8066b26a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/762d1b01956f4dcab60292bc8066b26a 2023-03-31 18:54:46,611 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/762d1b01956f4dcab60292bc8066b26a, entries=100, sequenceid=522, filesize=9.5 K 2023-03-31 18:54:46,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 8fda10a48807d71ff04525ce9d36a129 in 1657ms, sequenceid=522, compaction requested=false 2023-03-31 18:54:46,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9202414471d3472286be797514a9a738, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/008ed16571eb439f81b063e435f98a00, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0148ddcd4348443fb9eea59ff71842fc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/4f2ab3c174754301b1503c833e32b048, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a513c1c7ec9b487097524285209ef778, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/b0a81ab8d0a942738c5d2412916abd2a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0dea6dfbb3c848828fdf393e9f89df62, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/1124ed502f064594823b1bfa25c9fda6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0cb5b300305c4ba6b6af87b17761a4f6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ad788f90569242a4a12e6cc7d00da7f5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ee94c2a734f849b49b9f6e1adc375bcd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/070fc876f1604cbb9340607a41add227, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/c7cbcabb2f814ae2b21e67660493f0db, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/18ac124e5d724c52a967e10ed8269aef, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a0683241e09f4b97b7ef9b2226365481, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/77ae87e378574405a20e9b717cbd5521, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9c1da7fbe96845aca57b1df339a489ab, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/7889825a6e664eed8a87bedc5938be6b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/67f7cc09a34e4daca5902cf0e5861c19, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/41952185611f4769becf869a7045da20, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/723b130b5ad74236a94bf6cfc77ba376, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/25eac5d5f3ac4144b859a3a2d5ffceab, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9a5df6fad66d439a9ef881456157ff84] to archive 2023-03-31 18:54:46,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:54:46,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9202414471d3472286be797514a9a738 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9202414471d3472286be797514a9a738 2023-03-31 18:54:46,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/008ed16571eb439f81b063e435f98a00 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/008ed16571eb439f81b063e435f98a00 2023-03-31 18:54:46,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0148ddcd4348443fb9eea59ff71842fc to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0148ddcd4348443fb9eea59ff71842fc 2023-03-31 18:54:46,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/4f2ab3c174754301b1503c833e32b048 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/4f2ab3c174754301b1503c833e32b048 2023-03-31 18:54:46,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a513c1c7ec9b487097524285209ef778 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a513c1c7ec9b487097524285209ef778 2023-03-31 18:54:46,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/b0a81ab8d0a942738c5d2412916abd2a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/b0a81ab8d0a942738c5d2412916abd2a 2023-03-31 18:54:46,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0dea6dfbb3c848828fdf393e9f89df62 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0dea6dfbb3c848828fdf393e9f89df62 2023-03-31 18:54:46,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/1124ed502f064594823b1bfa25c9fda6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/1124ed502f064594823b1bfa25c9fda6 2023-03-31 18:54:46,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0cb5b300305c4ba6b6af87b17761a4f6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/0cb5b300305c4ba6b6af87b17761a4f6 2023-03-31 18:54:46,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ad788f90569242a4a12e6cc7d00da7f5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ad788f90569242a4a12e6cc7d00da7f5 2023-03-31 18:54:46,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ee94c2a734f849b49b9f6e1adc375bcd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/ee94c2a734f849b49b9f6e1adc375bcd 2023-03-31 18:54:46,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/070fc876f1604cbb9340607a41add227 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/070fc876f1604cbb9340607a41add227 2023-03-31 18:54:46,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/c7cbcabb2f814ae2b21e67660493f0db to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/c7cbcabb2f814ae2b21e67660493f0db 2023-03-31 18:54:46,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/18ac124e5d724c52a967e10ed8269aef to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/18ac124e5d724c52a967e10ed8269aef 2023-03-31 18:54:46,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a0683241e09f4b97b7ef9b2226365481 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/a0683241e09f4b97b7ef9b2226365481 2023-03-31 18:54:46,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/77ae87e378574405a20e9b717cbd5521 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/77ae87e378574405a20e9b717cbd5521 2023-03-31 18:54:46,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9c1da7fbe96845aca57b1df339a489ab to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9c1da7fbe96845aca57b1df339a489ab 2023-03-31 18:54:46,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/7889825a6e664eed8a87bedc5938be6b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/7889825a6e664eed8a87bedc5938be6b 2023-03-31 18:54:46,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/67f7cc09a34e4daca5902cf0e5861c19 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/67f7cc09a34e4daca5902cf0e5861c19 2023-03-31 18:54:46,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/41952185611f4769becf869a7045da20 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/41952185611f4769becf869a7045da20 2023-03-31 18:54:46,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/723b130b5ad74236a94bf6cfc77ba376 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/723b130b5ad74236a94bf6cfc77ba376 2023-03-31 18:54:46,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/25eac5d5f3ac4144b859a3a2d5ffceab to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/25eac5d5f3ac4144b859a3a2d5ffceab 2023-03-31 18:54:46,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9a5df6fad66d439a9ef881456157ff84 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/9a5df6fad66d439a9ef881456157ff84 2023-03-31 18:54:46,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/a660aa2deda14b6d9be62e9d440ab241, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/17e0e270cf244e70837e6fd067a8a818, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/36cb8fab67314bf9a44b87f50b5847de, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/22921ed69f054213955be3d2bca6dbf4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/55dde17050b64f2aad0bf36c57286f3c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/d5e7da09a84a4811b94d27cd9dbc376b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e20eaa4d880a4464a830b9151aa2ae79, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/9ff667eb30e54c54bbcbb81b12e515d4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/56173d0709d7430384e102a3f2947d92, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/f44c406257924ac6b077a37992146e47, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7fa125b2cf87464591c6ec848e174d46, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/ae35692b5bf44904804b697c6dba7af5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/43c115ada62943a1931584c46efb9d86, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/6f1c8702e5464c6db1f079be34335d50, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/012337e253ef42398b2e8c5e87d5cb31, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8d270656207741cdba1aa522d85a6a7f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7707369c4b524d70820e342e9e00a840, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8369abd9276a4388bf859ee16c1e29ea, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/2fdaa3a5489a420da60649e4ae0c0e9b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/543cb013bd9941dd9b1eb8bd35ef1243, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/5b1e4e590b844884891081c902b1c35b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3a036293b8845d19fc7d19bcb262ef7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3b9694b8ac74f3fade66e7327540eeb] to archive 2023-03-31 18:54:46,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:54:46,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/a660aa2deda14b6d9be62e9d440ab241 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/a660aa2deda14b6d9be62e9d440ab241 2023-03-31 18:54:46,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/17e0e270cf244e70837e6fd067a8a818 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/17e0e270cf244e70837e6fd067a8a818 2023-03-31 18:54:46,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/36cb8fab67314bf9a44b87f50b5847de to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/36cb8fab67314bf9a44b87f50b5847de 2023-03-31 18:54:46,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/22921ed69f054213955be3d2bca6dbf4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/22921ed69f054213955be3d2bca6dbf4 2023-03-31 18:54:46,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/55dde17050b64f2aad0bf36c57286f3c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/55dde17050b64f2aad0bf36c57286f3c 2023-03-31 18:54:46,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/d5e7da09a84a4811b94d27cd9dbc376b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/d5e7da09a84a4811b94d27cd9dbc376b 2023-03-31 18:54:46,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e20eaa4d880a4464a830b9151aa2ae79 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e20eaa4d880a4464a830b9151aa2ae79 2023-03-31 18:54:46,675 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/9ff667eb30e54c54bbcbb81b12e515d4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/9ff667eb30e54c54bbcbb81b12e515d4 2023-03-31 18:54:46,676 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/56173d0709d7430384e102a3f2947d92 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/56173d0709d7430384e102a3f2947d92 2023-03-31 18:54:46,677 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/f44c406257924ac6b077a37992146e47 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/f44c406257924ac6b077a37992146e47 2023-03-31 18:54:46,679 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7fa125b2cf87464591c6ec848e174d46 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7fa125b2cf87464591c6ec848e174d46 2023-03-31 18:54:46,680 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/ae35692b5bf44904804b697c6dba7af5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/ae35692b5bf44904804b697c6dba7af5 2023-03-31 18:54:46,681 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/43c115ada62943a1931584c46efb9d86 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/43c115ada62943a1931584c46efb9d86 2023-03-31 18:54:46,682 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/6f1c8702e5464c6db1f079be34335d50 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/6f1c8702e5464c6db1f079be34335d50 2023-03-31 18:54:46,683 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/012337e253ef42398b2e8c5e87d5cb31 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/012337e253ef42398b2e8c5e87d5cb31 2023-03-31 18:54:46,684 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8d270656207741cdba1aa522d85a6a7f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8d270656207741cdba1aa522d85a6a7f 2023-03-31 18:54:46,685 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7707369c4b524d70820e342e9e00a840 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/7707369c4b524d70820e342e9e00a840 2023-03-31 18:54:46,686 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8369abd9276a4388bf859ee16c1e29ea to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/8369abd9276a4388bf859ee16c1e29ea 2023-03-31 18:54:46,687 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/2fdaa3a5489a420da60649e4ae0c0e9b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/2fdaa3a5489a420da60649e4ae0c0e9b 2023-03-31 18:54:46,688 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/543cb013bd9941dd9b1eb8bd35ef1243 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/543cb013bd9941dd9b1eb8bd35ef1243 2023-03-31 18:54:46,689 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/5b1e4e590b844884891081c902b1c35b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/5b1e4e590b844884891081c902b1c35b 2023-03-31 18:54:46,690 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3a036293b8845d19fc7d19bcb262ef7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3a036293b8845d19fc7d19bcb262ef7 2023-03-31 18:54:46,691 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3b9694b8ac74f3fade66e7327540eeb to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/e3b9694b8ac74f3fade66e7327540eeb 2023-03-31 18:54:46,693 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/1d3cd3decac7496c82d32488a1aebf4a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2753d0ca6e74429bb48a5627eead58c1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d585de531f74c0197253ec47aff3f15, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/be0494c20f1243e3b57287a35c39c561, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/065a3d45db574555adc008f6ac2b0b74, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/3a087997f23f4558b444b2eb1c95138d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/a26f40721fed449bb3906b553bee29a1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/0d422e4536754976ab48e09cbf35865d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/d55046dd82b24ace908be96a1cb1544a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e95fcb9d8ad94ea99bfa1fcba6e06ca5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e7dc4248870d4591879d3c29f828af69, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d7b7f83cc58484e857f5f5a84f0f91a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/ff5054b33fc94526af5b7ab63dc2bf7c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/f4748cab75d84dbbb896eca70bbbd22d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/70e7204ea83d44c8adaaeb091cde784a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/aad72891c85e4aa2ad6a648db10c4717, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2b70f532d2cc4a5a82acc4d28ba72c54, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/eececc4a8e9a4bebbe2d7f42f57d6b58, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/df4e8762cd03444493ef83562ae68350, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/b8ff6d58e8164c1e820e9340e2d71d4e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/9c7af872a65f46378b092e702d00b8c5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/7e1e23b4b793436f8e555a026a017768] to archive 2023-03-31 18:54:46,694 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:54:46,695 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/1d3cd3decac7496c82d32488a1aebf4a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/1d3cd3decac7496c82d32488a1aebf4a 2023-03-31 18:54:46,696 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2753d0ca6e74429bb48a5627eead58c1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2753d0ca6e74429bb48a5627eead58c1 2023-03-31 18:54:46,697 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d585de531f74c0197253ec47aff3f15 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d585de531f74c0197253ec47aff3f15 2023-03-31 18:54:46,698 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/be0494c20f1243e3b57287a35c39c561 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/be0494c20f1243e3b57287a35c39c561 2023-03-31 18:54:46,698 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/065a3d45db574555adc008f6ac2b0b74 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/065a3d45db574555adc008f6ac2b0b74 2023-03-31 18:54:46,699 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/3a087997f23f4558b444b2eb1c95138d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/3a087997f23f4558b444b2eb1c95138d 2023-03-31 18:54:46,700 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/a26f40721fed449bb3906b553bee29a1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/a26f40721fed449bb3906b553bee29a1 2023-03-31 18:54:46,701 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/0d422e4536754976ab48e09cbf35865d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/0d422e4536754976ab48e09cbf35865d 2023-03-31 18:54:46,702 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/d55046dd82b24ace908be96a1cb1544a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/d55046dd82b24ace908be96a1cb1544a 2023-03-31 18:54:46,703 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e95fcb9d8ad94ea99bfa1fcba6e06ca5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e95fcb9d8ad94ea99bfa1fcba6e06ca5 2023-03-31 18:54:46,704 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e7dc4248870d4591879d3c29f828af69 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/e7dc4248870d4591879d3c29f828af69 2023-03-31 18:54:46,705 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d7b7f83cc58484e857f5f5a84f0f91a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/6d7b7f83cc58484e857f5f5a84f0f91a 2023-03-31 18:54:46,706 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/ff5054b33fc94526af5b7ab63dc2bf7c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/ff5054b33fc94526af5b7ab63dc2bf7c 2023-03-31 18:54:46,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/f4748cab75d84dbbb896eca70bbbd22d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/f4748cab75d84dbbb896eca70bbbd22d 2023-03-31 18:54:46,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/70e7204ea83d44c8adaaeb091cde784a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/70e7204ea83d44c8adaaeb091cde784a 2023-03-31 18:54:46,708 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/aad72891c85e4aa2ad6a648db10c4717 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/aad72891c85e4aa2ad6a648db10c4717 2023-03-31 18:54:46,709 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2b70f532d2cc4a5a82acc4d28ba72c54 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/2b70f532d2cc4a5a82acc4d28ba72c54 2023-03-31 18:54:46,710 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/eececc4a8e9a4bebbe2d7f42f57d6b58 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/eececc4a8e9a4bebbe2d7f42f57d6b58 2023-03-31 18:54:46,711 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/df4e8762cd03444493ef83562ae68350 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/df4e8762cd03444493ef83562ae68350 2023-03-31 18:54:46,711 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/b8ff6d58e8164c1e820e9340e2d71d4e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/b8ff6d58e8164c1e820e9340e2d71d4e 2023-03-31 18:54:46,712 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/9c7af872a65f46378b092e702d00b8c5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/9c7af872a65f46378b092e702d00b8c5 2023-03-31 18:54:46,713 DEBUG [StoreCloser-TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/7e1e23b4b793436f8e555a026a017768 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/7e1e23b4b793436f8e555a026a017768 2023-03-31 18:54:46,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/recovered.edits/525.seqid, newMaxSeqId=525, maxSeqId=4 2023-03-31 18:54:46,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129. 2023-03-31 18:54:46,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8fda10a48807d71ff04525ce9d36a129: 2023-03-31 18:54:46,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:46,720 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=8fda10a48807d71ff04525ce9d36a129, regionState=CLOSED 2023-03-31 18:54:46,720 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288886720"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288886720"}]},"ts":"1680288886720"} 2023-03-31 18:54:46,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-31 18:54:46,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 8fda10a48807d71ff04525ce9d36a129, server=jenkins-hbase4.apache.org,34951,1680288816339 in 1.9180 sec 2023-03-31 18:54:46,723 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-31 18:54:46,723 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8fda10a48807d71ff04525ce9d36a129, UNASSIGN in 1.9220 sec 2023-03-31 18:54:46,724 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288886724"}]},"ts":"1680288886724"} 2023-03-31 18:54:46,725 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 18:54:46,727 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 18:54:46,729 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9350 sec 2023-03-31 18:54:46,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 18:54:46,900 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-31 18:54:46,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 18:54:46,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:46,903 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:46,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-31 18:54:46,904 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:46,909 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:46,912 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/recovered.edits] 2023-03-31 18:54:46,915 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/2801395e237c4ce3bd60b2db77964cc2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/2801395e237c4ce3bd60b2db77964cc2 2023-03-31 18:54:46,917 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/dac6ed4f3ee346088712d41073355ff3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/A/dac6ed4f3ee346088712d41073355ff3 2023-03-31 18:54:46,921 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/3f8e0c34cb14459a9c888855ca28bd07 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/3f8e0c34cb14459a9c888855ca28bd07 2023-03-31 18:54:46,923 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/78cb9670786449c1ac3485b185f665dd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/B/78cb9670786449c1ac3485b185f665dd 2023-03-31 18:54:46,925 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/762d1b01956f4dcab60292bc8066b26a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/762d1b01956f4dcab60292bc8066b26a 2023-03-31 18:54:46,927 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/bfee61f266ba497187e65d301256db40 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/C/bfee61f266ba497187e65d301256db40 2023-03-31 18:54:46,930 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/recovered.edits/525.seqid to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129/recovered.edits/525.seqid 2023-03-31 18:54:46,930 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/8fda10a48807d71ff04525ce9d36a129 2023-03-31 18:54:46,930 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:54:46,931 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 18:54:46,932 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-31 18:54:46,936 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311078e8f68777469eb89c02ba78fadd64 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311078e8f68777469eb89c02ba78fadd64 2023-03-31 18:54:46,937 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311eec44c476474ccf92ea6e16d7703130 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311eec44c476474ccf92ea6e16d7703130 2023-03-31 18:54:46,938 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311f68badeabd645568e0a18400faa71dd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311f68badeabd645568e0a18400faa71dd 2023-03-31 18:54:46,940 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312f3f1ef6944d4310aa992e1ee5e4a426 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312f3f1ef6944d4310aa992e1ee5e4a426 2023-03-31 18:54:46,941 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313069b4a068ed45fc961141b9412c7d23 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313069b4a068ed45fc961141b9412c7d23 2023-03-31 18:54:46,943 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033136bf2e8eab8b45d4bb8e630ead2ea87d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033136bf2e8eab8b45d4bb8e630ead2ea87d 2023-03-31 18:54:46,944 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313aee8ab940ce401ea071a22f9b97b61a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313aee8ab940ce401ea071a22f9b97b61a 2023-03-31 18:54:46,945 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315aa2227ec4fa492888b5132bdb476498 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315aa2227ec4fa492888b5132bdb476498 2023-03-31 18:54:46,946 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033160a581c88a2c4210aec76921c6326a96 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033160a581c88a2c4210aec76921c6326a96 2023-03-31 18:54:46,947 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318346391e559145588e3081ca68e6bb8f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318346391e559145588e3081ca68e6bb8f 2023-03-31 18:54:46,949 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033187b9d766d9e148de8a5048b5f035bbd5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033187b9d766d9e148de8a5048b5f035bbd5 2023-03-31 18:54:46,950 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033193ed7b1dbd8547f49e00f17fdd015397 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033193ed7b1dbd8547f49e00f17fdd015397 2023-03-31 18:54:46,951 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319b824ff9cb7447e7b1b5c71bf87e5ec9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319b824ff9cb7447e7b1b5c71bf87e5ec9 2023-03-31 18:54:46,952 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a64807842b514da8b5534840baf63389 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a64807842b514da8b5534840baf63389 2023-03-31 18:54:46,953 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a91796280d694db692b3e67e6a0c7047 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a91796280d694db692b3e67e6a0c7047 2023-03-31 18:54:46,954 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b2afbe7f6837461d8b53ba8f436efab4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b2afbe7f6837461d8b53ba8f436efab4 2023-03-31 18:54:46,954 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c36fa9dd9d10407b8774c427823afead to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c36fa9dd9d10407b8774c427823afead 2023-03-31 18:54:46,955 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e9d6a59e3053488992d1cc838fc56f30 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e9d6a59e3053488992d1cc838fc56f30 2023-03-31 18:54:46,956 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f10fd65bfce941e899d71c354e732ea9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f10fd65bfce941e899d71c354e732ea9 2023-03-31 18:54:46,957 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f1ce28daee624f528f83d4ddd5078de6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f1ce28daee624f528f83d4ddd5078de6 2023-03-31 18:54:46,958 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 18:54:46,960 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:46,961 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 18:54:46,963 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 18:54:46,965 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:46,965 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 18:54:46,965 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680288886965"}]},"ts":"9223372036854775807"} 2023-03-31 18:54:46,966 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 18:54:46,967 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8fda10a48807d71ff04525ce9d36a129, NAME => 'TestAcidGuarantees,,1680288853212.8fda10a48807d71ff04525ce9d36a129.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 18:54:46,967 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 18:54:46,967 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680288886967"}]},"ts":"9223372036854775807"} 2023-03-31 18:54:46,968 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 18:54:46,970 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:46,971 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 69 msec 2023-03-31 18:54:47,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-31 18:54:47,004 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-31 18:54:47,016 INFO [Listener at localhost/43327] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 231), OpenFileDescriptor=515 (was 515), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=351 (was 344) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3640 (was 3803) 2023-03-31 18:54:47,027 INFO [Listener at localhost/43327] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=515, MaxFileDescriptor=60000, SystemLoadAverage=351, ProcessCount=169, AvailableMemoryMB=3639 2023-03-31 18:54:47,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 18:54:47,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 18:54:47,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 18:54:47,031 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 18:54:47,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-31 18:54:47,032 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 18:54:47,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 18:54:47,033 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,034 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898 empty. 2023-03-31 18:54:47,034 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,034 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:54:47,042 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 18:54:47,044 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2b9d344a6097fb5c7c41dc8319fac898, NAME => 'TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp 2023-03-31 18:54:47,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 18:54:47,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 18:54:47,449 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:54:47,449 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2b9d344a6097fb5c7c41dc8319fac898, disabling compactions & flushes 2023-03-31 18:54:47,449 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:47,449 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:47,449 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. after waiting 0 ms 2023-03-31 18:54:47,449 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:47,449 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:47,450 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:47,452 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 18:54:47,452 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288887452"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288887452"}]},"ts":"1680288887452"} 2023-03-31 18:54:47,453 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 18:54:47,454 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 18:54:47,454 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288887454"}]},"ts":"1680288887454"} 2023-03-31 18:54:47,455 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 18:54:47,457 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b9d344a6097fb5c7c41dc8319fac898, ASSIGN}] 2023-03-31 18:54:47,459 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b9d344a6097fb5c7c41dc8319fac898, ASSIGN 2023-03-31 18:54:47,459 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b9d344a6097fb5c7c41dc8319fac898, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34951,1680288816339; forceNewPlan=false, retain=false 2023-03-31 18:54:47,610 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=2b9d344a6097fb5c7c41dc8319fac898, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:47,610 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288887610"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288887610"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288887610"}]},"ts":"1680288887610"} 2023-03-31 18:54:47,612 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:54:47,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 18:54:47,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:47,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2b9d344a6097fb5c7c41dc8319fac898, NAME => 'TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:54:47,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:54:47,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,768 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,769 DEBUG [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A 2023-03-31 18:54:47,769 DEBUG [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A 2023-03-31 18:54:47,770 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:54:47,770 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b9d344a6097fb5c7c41dc8319fac898 columnFamilyName A 2023-03-31 18:54:47,770 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] regionserver.HStore(310): Store=2b9d344a6097fb5c7c41dc8319fac898/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:54:47,770 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,771 DEBUG [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B 2023-03-31 18:54:47,772 DEBUG [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B 2023-03-31 18:54:47,772 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:54:47,772 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b9d344a6097fb5c7c41dc8319fac898 columnFamilyName B 2023-03-31 18:54:47,772 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] regionserver.HStore(310): Store=2b9d344a6097fb5c7c41dc8319fac898/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:54:47,772 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,773 DEBUG [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C 2023-03-31 18:54:47,773 DEBUG [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C 2023-03-31 18:54:47,774 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:54:47,774 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b9d344a6097fb5c7c41dc8319fac898 columnFamilyName C 2023-03-31 18:54:47,774 INFO [StoreOpener-2b9d344a6097fb5c7c41dc8319fac898-1] regionserver.HStore(310): Store=2b9d344a6097fb5c7c41dc8319fac898/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:54:47,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:47,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:54:47,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:47,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 18:54:47,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2b9d344a6097fb5c7c41dc8319fac898; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70969681, jitterRate=0.05753065645694733}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:54:47,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:47,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., pid=30, masterSystemTime=1680288887763 2023-03-31 18:54:47,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:47,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:47,782 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=2b9d344a6097fb5c7c41dc8319fac898, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:47,782 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288887782"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680288887782"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680288887782"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680288887782"}]},"ts":"1680288887782"} 2023-03-31 18:54:47,784 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-31 18:54:47,784 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 in 172 msec 2023-03-31 18:54:47,785 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-31 18:54:47,785 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b9d344a6097fb5c7c41dc8319fac898, ASSIGN in 327 msec 2023-03-31 18:54:47,786 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 18:54:47,786 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288887786"}]},"ts":"1680288887786"} 2023-03-31 18:54:47,787 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 18:54:47,790 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 18:54:47,791 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 761 msec 2023-03-31 18:54:48,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 18:54:48,136 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-31 18:54:48,136 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,141 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,142 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,143 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52492, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,144 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 18:54:48,146 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48932, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 18:54:48,147 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,151 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,151 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,154 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,155 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,160 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,160 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,163 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,163 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x7810a231 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,167 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104dbf4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,167 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,170 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,170 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,174 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,174 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,179 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,179 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,184 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,185 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x2de064cb to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:54:48,188 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f8ff99f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:54:48,194 DEBUG [hconnection-0x5a76c1b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,194 DEBUG [hconnection-0x624c369d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,195 DEBUG [hconnection-0x76e2503d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,195 DEBUG [hconnection-0x77a73ed0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:48,195 DEBUG [hconnection-0xfff38f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,196 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52518, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,197 DEBUG [hconnection-0x1bcaaf58-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,197 DEBUG [hconnection-0x39be9191-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,198 DEBUG [hconnection-0x3be52075-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,198 DEBUG [hconnection-0x5aba5340-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,198 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,200 DEBUG [hconnection-0x2564ed01-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:54:48,201 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52584, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,202 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,202 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52544, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,202 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52570, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,210 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,210 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52530, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,211 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52554, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,211 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52590, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:54:48,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:48,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:48,212 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:48,212 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:48,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:48,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:48,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:48,218 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:54:48,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:48,220 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:48,224 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:48,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:48,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:48,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:48,242 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:48,242 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:48,242 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:48,242 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:48,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288948239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288948240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288948242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288948245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288948245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,252 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,253 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,253 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eba386d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:48,255 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:48,255 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,257 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:48,258 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,258 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:48,259 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,259 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,259 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:48,259 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:48,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:48,262 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:48,262 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:48,262 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:48,262 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:48,262 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:48,264 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,264 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,264 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:48,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:48,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/3f2adb2a28834915bf733ab6255b7b8c 2023-03-31 18:54:48,271 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,271 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,271 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,271 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:48,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:48,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:48,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,272 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:48,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:48,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:48,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:48,273 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:48,273 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:48,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:48,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/67755d4550224652ba8170476a90d351 2023-03-31 18:54:48,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288948345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288948346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288948346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288948347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288948348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,373 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:48,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:48,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:48,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:48,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:48,377 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:48,377 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:48,377 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:48,378 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:48,382 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,382 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,383 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,383 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:48,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,383 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,384 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:48,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:48,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:48,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:48,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:48,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:48,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:48,387 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53383cc7[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:48,389 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:48,389 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:48,390 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,390 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:48,393 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,393 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:48,393 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:48,393 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:48,395 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:48,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:48,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:48,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:48,395 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:48,397 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,397 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,397 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,398 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:48,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:48,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:48,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:48,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:48,402 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:48,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:48,402 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:48,402 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:48,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:48,502 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:48,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:48,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:48,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:48,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:48,506 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:48,506 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:48,506 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:48,506 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:48,506 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,506 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:48,511 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:48,512 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,512 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:48,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:48,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:48,514 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:48,514 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:48,514 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:48,514 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:48,514 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:48,514 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:48,514 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,514 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:48,514 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:48,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,516 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,516 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,516 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:48,529 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:48,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@354a6776[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:48,529 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,530 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,530 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:48,530 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,530 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:48,530 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:48,530 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:48,532 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,532 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,532 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:48,532 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:48,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:48,534 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:48,534 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:48,534 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:48,534 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:48,534 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:48,536 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,536 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,536 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:48,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,537 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,542 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:48,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:48,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:48,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:48,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:48,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:48,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:48,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:48,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,543 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,543 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,543 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:48,543 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:48,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:48,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288948547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288948549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288948549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288948552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288948553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,643 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:48,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:48,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:48,646 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:48,646 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:48,646 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:48,646 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:48,647 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,647 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,648 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,648 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:48,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:48,648 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,648 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:48,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:48,650 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:48,650 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:48,650 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:48,650 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:48,653 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:48,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6445f965[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:48,654 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:48,655 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:48,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:48,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:48,656 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,656 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:48,657 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:48,657 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:48,658 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:48,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:48,666 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:48,666 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:48,666 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:48,666 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:48,666 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:48,668 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,668 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:48,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,668 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:48,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:48,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,670 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:48,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:48,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:48,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:48,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:48,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:48,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:48,673 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:48,673 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:48,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:48,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/389cb735f82a4ef7bc6e56c9586ffb4c 2023-03-31 18:54:48,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/3f2adb2a28834915bf733ab6255b7b8c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/3f2adb2a28834915bf733ab6255b7b8c 2023-03-31 18:54:48,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/3f2adb2a28834915bf733ab6255b7b8c, entries=150, sequenceid=13, filesize=11.6 K 2023-03-31 18:54:48,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/67755d4550224652ba8170476a90d351 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/67755d4550224652ba8170476a90d351 2023-03-31 18:54:48,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/67755d4550224652ba8170476a90d351, entries=150, sequenceid=13, filesize=11.6 K 2023-03-31 18:54:48,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/389cb735f82a4ef7bc6e56c9586ffb4c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/389cb735f82a4ef7bc6e56c9586ffb4c 2023-03-31 18:54:48,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/389cb735f82a4ef7bc6e56c9586ffb4c, entries=150, sequenceid=13, filesize=11.6 K 2023-03-31 18:54:48,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2b9d344a6097fb5c7c41dc8319fac898 in 525ms, sequenceid=13, compaction requested=false 2023-03-31 18:54:48,743 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:48,773 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:48,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:48,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:48,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:48,777 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:48,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:48,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:48,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:48,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,778 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:48,778 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:48,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:48,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:48,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:48,781 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:48,782 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:48,782 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:48,782 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:48,800 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/bdf74fa30c0346a2a5d7ca41e781ccb7 2023-03-31 18:54:48,820 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/2f7f8e2c11da4f638e7ec327b7d18a5d 2023-03-31 18:54:48,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:48,853 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:48,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288948858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288948860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288948861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288948861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288948862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288948962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288948964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288948965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288948965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:48,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:48,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288948965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288949164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288949166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288949167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288949167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288949168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,238 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/d7a8726ddc244f6dbdf5a4c01c6f0901 2023-03-31 18:54:49,244 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/bdf74fa30c0346a2a5d7ca41e781ccb7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/bdf74fa30c0346a2a5d7ca41e781ccb7 2023-03-31 18:54:49,248 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/bdf74fa30c0346a2a5d7ca41e781ccb7, entries=150, sequenceid=33, filesize=11.6 K 2023-03-31 18:54:49,249 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/2f7f8e2c11da4f638e7ec327b7d18a5d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2f7f8e2c11da4f638e7ec327b7d18a5d 2023-03-31 18:54:49,253 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2f7f8e2c11da4f638e7ec327b7d18a5d, entries=150, sequenceid=33, filesize=11.6 K 2023-03-31 18:54:49,253 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/d7a8726ddc244f6dbdf5a4c01c6f0901 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d7a8726ddc244f6dbdf5a4c01c6f0901 2023-03-31 18:54:49,257 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d7a8726ddc244f6dbdf5a4c01c6f0901, entries=150, sequenceid=33, filesize=11.6 K 2023-03-31 18:54:49,258 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 2b9d344a6097fb5c7c41dc8319fac898 in 477ms, sequenceid=33, compaction requested=false 2023-03-31 18:54:49,258 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:49,258 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:49,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:49,260 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:49,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:49,260 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:49,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:49,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bc3ccd0[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:49,262 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:49,262 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:49,265 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:49,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:49,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:49,266 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:49,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:49,270 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:49,270 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:49,270 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:49,270 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:49,270 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:49,271 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:49,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:49,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:49,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:49,272 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:49,273 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,276 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,276 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:49,276 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:49,276 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:49,276 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:49,276 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:49,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:49,276 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:49,277 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:49,277 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:49,277 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:49,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:49,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:49,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:49,277 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:49,277 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:49,377 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:49,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:49,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:49,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:49,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:49,380 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:49,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:49,381 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:49,381 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:49,381 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:49,381 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:49,383 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:49,383 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:49,384 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,384 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:49,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:49,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:49,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:49,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:49,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:49,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:49,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:49,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:49,385 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:49,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:49,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:49,385 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 18:54:49,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:49,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:49,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:49,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:49,385 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:49,386 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:49,396 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/a8f4758df97c42b5b4909d8207f1a605 2023-03-31 18:54:49,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:49,470 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:49,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288949480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288949481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288949483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288949484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288949484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288949585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288949585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288949587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288949589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288949589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288949787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288949787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288949790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288949791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:49,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288949791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:49,810 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/b2295ef6bf4b4569a3c441a17e75edcf 2023-03-31 18:54:50,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288950090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288950090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288950095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288950095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288950096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,227 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/291a9634ca0d4914b19ee11ae56c442e 2023-03-31 18:54:50,232 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/a8f4758df97c42b5b4909d8207f1a605 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a8f4758df97c42b5b4909d8207f1a605 2023-03-31 18:54:50,242 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a8f4758df97c42b5b4909d8207f1a605, entries=150, sequenceid=46, filesize=11.6 K 2023-03-31 18:54:50,243 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/b2295ef6bf4b4569a3c441a17e75edcf as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b2295ef6bf4b4569a3c441a17e75edcf 2023-03-31 18:54:50,248 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b2295ef6bf4b4569a3c441a17e75edcf, entries=150, sequenceid=46, filesize=11.6 K 2023-03-31 18:54:50,249 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/291a9634ca0d4914b19ee11ae56c442e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/291a9634ca0d4914b19ee11ae56c442e 2023-03-31 18:54:50,253 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/291a9634ca0d4914b19ee11ae56c442e, entries=150, sequenceid=46, filesize=11.6 K 2023-03-31 18:54:50,254 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2b9d344a6097fb5c7c41dc8319fac898 in 869ms, sequenceid=46, compaction requested=true 2023-03-31 18:54:50,254 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:50,254 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:50,254 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:50,254 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:50,254 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:50,257 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,257 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:50,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:50,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@98f156b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:50,259 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:50,259 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:50,261 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:50,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:50,261 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:50,261 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:50,261 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:50,261 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,261 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:50,263 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:50,263 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,263 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:50,263 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:50,285 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:50,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:50,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:50,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:50,285 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:50,288 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:50,288 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:50,288 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:50,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:50,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:50,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:50,291 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:50,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:50,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:50,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:50,293 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:50,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:50,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:50,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:50,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:50,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:50,293 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:50,294 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:50,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:50,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:50,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:50,394 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:50,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:50,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:50,397 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:50,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:50,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:50,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:50,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:50,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:50,398 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,398 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:50,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:50,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:50,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:50,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:50,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:50,400 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:50,400 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:50,400 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-31 18:54:50,401 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:50,401 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:50,401 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:50,401 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:50,401 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:50,401 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:50,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:50,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:50,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288950604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288950604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288950605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288950607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288950607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288950708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288950709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288950711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288950711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,811 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/102467ed292f418eb8f8b6db5fa39c7e 2023-03-31 18:54:50,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288950912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288950912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288950914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:50,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:50,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288950915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288951216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288951217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,222 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/6678582d3531409bbd189ff0213715c7 2023-03-31 18:54:51,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288951221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288951222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,240 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/112e2a8201cc4b92beaaa22006b3b781 2023-03-31 18:54:51,245 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/102467ed292f418eb8f8b6db5fa39c7e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/102467ed292f418eb8f8b6db5fa39c7e 2023-03-31 18:54:51,250 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/102467ed292f418eb8f8b6db5fa39c7e, entries=150, sequenceid=65, filesize=11.6 K 2023-03-31 18:54:51,251 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/6678582d3531409bbd189ff0213715c7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/6678582d3531409bbd189ff0213715c7 2023-03-31 18:54:51,260 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/6678582d3531409bbd189ff0213715c7, entries=150, sequenceid=65, filesize=11.6 K 2023-03-31 18:54:51,261 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/112e2a8201cc4b92beaaa22006b3b781 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/112e2a8201cc4b92beaaa22006b3b781 2023-03-31 18:54:51,272 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/112e2a8201cc4b92beaaa22006b3b781, entries=150, sequenceid=65, filesize=11.6 K 2023-03-31 18:54:51,272 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2b9d344a6097fb5c7c41dc8319fac898 in 872ms, sequenceid=65, compaction requested=true 2023-03-31 18:54:51,273 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:51,273 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:51,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:51,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:51,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:51,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:51,274 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,274 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f98a2fb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:51,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:51,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:51,278 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:51,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:51,278 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,278 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:51,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:51,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:51,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:51,282 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:51,282 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:51,282 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:51,283 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:51,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:51,292 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:51,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:51,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:51,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:51,292 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:51,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:51,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:51,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:51,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:51,295 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:51,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:51,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:51,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:51,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:51,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:51,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:51,301 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:51,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:51,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:51,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:51,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:51,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:51,301 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:51,302 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:51,402 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:51,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:51,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:51,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:51,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:51,405 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:51,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:51,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:51,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:51,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:51,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:51,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,407 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:51,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:51,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:51,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:51,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:51,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:51,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:51,412 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:51,412 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:51,412 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:51,416 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:51,416 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:51,416 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 18:54:51,416 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:51,416 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:51,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:51,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:51,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:51,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:51,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:51,620 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:51,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288951677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288951720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288951721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288951740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288951740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288951780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:51,830 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/6648d603818b447c8c249d8dd33e6807 2023-03-31 18:54:51,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:51,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288951985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,241 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/4a14cbc80e18486f902d586cbcc75024 2023-03-31 18:54:52,254 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/43550e95942d42ab8f27c3b05f4bc669 2023-03-31 18:54:52,259 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/6648d603818b447c8c249d8dd33e6807 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/6648d603818b447c8c249d8dd33e6807 2023-03-31 18:54:52,260 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 18:54:52,263 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/6648d603818b447c8c249d8dd33e6807, entries=150, sequenceid=78, filesize=11.6 K 2023-03-31 18:54:52,264 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/4a14cbc80e18486f902d586cbcc75024 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/4a14cbc80e18486f902d586cbcc75024 2023-03-31 18:54:52,276 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/4a14cbc80e18486f902d586cbcc75024, entries=150, sequenceid=78, filesize=11.6 K 2023-03-31 18:54:52,279 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/43550e95942d42ab8f27c3b05f4bc669 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/43550e95942d42ab8f27c3b05f4bc669 2023-03-31 18:54:52,290 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/43550e95942d42ab8f27c3b05f4bc669, entries=150, sequenceid=78, filesize=11.6 K 2023-03-31 18:54:52,291 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2b9d344a6097fb5c7c41dc8319fac898 in 875ms, sequenceid=78, compaction requested=true 2023-03-31 18:54:52,291 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:52,291 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:52,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:52,298 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,298 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,298 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,298 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:52,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24dde0bc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:52,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:52,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:52,303 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,304 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:52,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:52,305 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:54:52,305 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:52,305 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:52,305 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:52,305 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:52,305 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:52,305 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:52,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,306 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:52,306 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,306 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:52,310 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:52,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:52,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:52,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:52,310 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:52,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,313 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,313 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,314 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,316 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,316 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,316 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,316 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,317 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:52,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:52,317 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,317 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:52,317 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:52,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,317 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,317 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,317 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:52,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288952342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,417 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:52,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:52,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:52,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:52,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:52,421 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:52,421 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:52,421 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:52,421 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:52,421 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,421 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,422 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,422 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,423 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,423 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:52,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:52,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:52,425 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,425 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:52,425 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:52,425 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:52,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:52,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:52,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:52,427 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,427 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,427 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,427 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:52,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217d0dbb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:52,429 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:52,429 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,430 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:52,430 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,430 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:52,432 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:52,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:52,434 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:52,434 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:52,434 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:52,434 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:52,434 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:52,436 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,436 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,436 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,437 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,440 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,440 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,440 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,441 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:52,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:52,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,441 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:52,440 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,441 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:52,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:52,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288952445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,541 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:52,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:52,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:52,544 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:52,544 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:52,544 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:52,545 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:52,545 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,545 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,546 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,546 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,546 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,546 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:52,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:52,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:52,547 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:52,548 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,548 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:52,548 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:52,548 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:52,550 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,551 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:52,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a860b2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:52,553 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:52,553 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,554 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,554 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:52,554 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:52,554 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:52,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:52,556 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:52,556 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:52,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:52,559 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:52,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:52,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:52,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:52,559 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:52,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,561 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,562 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,566 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,566 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,566 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,566 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:52,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:52,566 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,566 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,567 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:52,567 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:52,566 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:52,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:52,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288952649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,667 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:52,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:52,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:52,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:52,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:52,671 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:52,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:52,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:52,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:52,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:52,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:52,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:52,675 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:52,675 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:52,675 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:52,675 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:52,675 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,675 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:52,675 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:52,675 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,675 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:52,677 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a0d8f26[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:52,679 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:52,680 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,681 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:52,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:52,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:52,681 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,681 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:52,682 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:52,685 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:52,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:52,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:52,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:52,685 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:52,687 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,687 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,687 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,688 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:52,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:52,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,692 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:52,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,692 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:52,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:52,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:52,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288952725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:52,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288952725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/86b0448d4da140058a9bc13c3a125d0d 2023-03-31 18:54:52,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:52,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288952748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:52,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288952749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,792 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:52,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:52,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:52,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:52,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:52,795 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:52,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:52,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:52,796 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:52,796 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,796 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,797 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,797 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,797 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:52,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:52,799 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,799 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:52,799 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:52,799 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:52,800 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,801 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,801 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:52,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:52,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311e48db[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:52,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,804 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:52,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:52,804 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,804 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:52,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:52,806 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:52,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:52,809 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:52,809 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:52,809 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:52,809 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:52,809 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:52,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,811 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,812 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,814 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:52,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:52,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:52,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,814 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:52,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,814 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:52,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,914 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:52,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:52,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:52,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:52,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:52,918 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:52,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:52,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:52,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:52,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,921 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,921 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,921 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,921 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:52,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:52,923 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,923 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:52,923 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:52,923 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:52,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,925 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,925 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,925 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:52,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c78b96[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:52,926 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:52,927 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,928 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:52,928 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:52,928 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:52,928 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,928 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:52,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:52,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:52,931 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:52,932 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:52,932 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:52,932 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:52,932 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:52,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:52,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:52,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,935 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,936 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:52,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:52,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:52,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:52,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:52,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:52,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:52,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:52,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:52,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:52,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:52,942 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:52,942 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:52,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:52,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288952954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,042 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:53,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:53,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:53,046 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:53,046 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,046 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:53,046 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:53,046 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,046 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,047 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:53,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:53,049 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,049 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:53,050 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:53,050 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:53,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:53,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:53,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:53,052 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,052 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,052 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,052 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf9c9cc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:53,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:53,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:53,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:53,055 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,055 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:53,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:53,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:53,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:53,059 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:53,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:53,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:53,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:53,059 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:53,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,061 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,063 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:53,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:53,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:53,066 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,067 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:53,067 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:53,067 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,067 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,149 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/c4914d23c199415dab2c7d9748f75332 2023-03-31 18:54:53,167 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:53,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:53,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/29daf27829e345538860ce84f6ed5f11 2023-03-31 18:54:53,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:53,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:53,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:53,173 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:53,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:53,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:53,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/86b0448d4da140058a9bc13c3a125d0d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/86b0448d4da140058a9bc13c3a125d0d 2023-03-31 18:54:53,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,176 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:53,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:53,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:53,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:53,180 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/86b0448d4da140058a9bc13c3a125d0d, entries=200, sequenceid=98, filesize=13.9 K 2023-03-31 18:54:53,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/c4914d23c199415dab2c7d9748f75332 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/c4914d23c199415dab2c7d9748f75332 2023-03-31 18:54:53,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:53,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:53,188 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,188 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:53,188 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:53,188 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:53,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:53,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:53,191 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,191 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:53,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d1b0f2a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:53,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:53,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,195 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:53,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:53,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:53,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:53,198 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/c4914d23c199415dab2c7d9748f75332, entries=150, sequenceid=98, filesize=11.6 K 2023-03-31 18:54:53,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/29daf27829e345538860ce84f6ed5f11 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/29daf27829e345538860ce84f6ed5f11 2023-03-31 18:54:53,201 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:53,201 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,201 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,201 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:53,206 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:53,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:53,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:53,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:53,206 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:53,209 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,209 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,209 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,211 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,212 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,217 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:53,217 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:53,217 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:53,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:53,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/29daf27829e345538860ce84f6ed5f11, entries=150, sequenceid=98, filesize=11.6 K 2023-03-31 18:54:53,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2b9d344a6097fb5c7c41dc8319fac898 in 917ms, sequenceid=98, compaction requested=true 2023-03-31 18:54:53,222 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:53,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:53,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:53,222 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 18:54:53,222 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 18:54:53,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:53,225 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 18:54:53,225 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/B is initiating minor compaction (all files) 2023-03-31 18:54:53,225 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/B in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,226 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/67755d4550224652ba8170476a90d351, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2f7f8e2c11da4f638e7ec327b7d18a5d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b2295ef6bf4b4569a3c441a17e75edcf, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/6678582d3531409bbd189ff0213715c7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/4a14cbc80e18486f902d586cbcc75024, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/c4914d23c199415dab2c7d9748f75332] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=69.6 K 2023-03-31 18:54:53,226 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73656 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 18:54:53,226 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/A is initiating minor compaction (all files) 2023-03-31 18:54:53,226 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/A in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,226 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/3f2adb2a28834915bf733ab6255b7b8c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/bdf74fa30c0346a2a5d7ca41e781ccb7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a8f4758df97c42b5b4909d8207f1a605, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/102467ed292f418eb8f8b6db5fa39c7e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/6648d603818b447c8c249d8dd33e6807, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/86b0448d4da140058a9bc13c3a125d0d] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=71.9 K 2023-03-31 18:54:53,226 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 67755d4550224652ba8170476a90d351, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680288888213 2023-03-31 18:54:53,227 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 3f2adb2a28834915bf733ab6255b7b8c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680288888213 2023-03-31 18:54:53,227 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting bdf74fa30c0346a2a5d7ca41e781ccb7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680288888226 2023-03-31 18:54:53,227 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2f7f8e2c11da4f638e7ec327b7d18a5d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680288888226 2023-03-31 18:54:53,228 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a8f4758df97c42b5b4909d8207f1a605, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680288888856 2023-03-31 18:54:53,228 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b2295ef6bf4b4569a3c441a17e75edcf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680288888856 2023-03-31 18:54:53,228 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 102467ed292f418eb8f8b6db5fa39c7e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680288889478 2023-03-31 18:54:53,229 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 6678582d3531409bbd189ff0213715c7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680288889478 2023-03-31 18:54:53,230 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 6648d603818b447c8c249d8dd33e6807, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680288890604 2023-03-31 18:54:53,230 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4a14cbc80e18486f902d586cbcc75024, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680288890604 2023-03-31 18:54:53,230 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 86b0448d4da140058a9bc13c3a125d0d, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680288891666 2023-03-31 18:54:53,230 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting c4914d23c199415dab2c7d9748f75332, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680288891671 2023-03-31 18:54:53,259 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#A#compaction#195 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:53,279 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#B#compaction#196 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:53,311 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/e97a507f776346e59e51e76ce313be00 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/e97a507f776346e59e51e76ce313be00 2023-03-31 18:54:53,313 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/9fe90f870d7d4181976dea3fd84b7ba2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9fe90f870d7d4181976dea3fd84b7ba2 2023-03-31 18:54:53,317 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:53,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:53,319 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/B of 2b9d344a6097fb5c7c41dc8319fac898 into e97a507f776346e59e51e76ce313be00(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:53,319 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:53,319 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/B, priority=10, startTime=1680288893222; duration=0sec 2023-03-31 18:54:53,319 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:53,319 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 18:54:53,320 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/A of 2b9d344a6097fb5c7c41dc8319fac898 into 9fe90f870d7d4181976dea3fd84b7ba2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:53,320 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:53,320 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/A, priority=10, startTime=1680288893222; duration=0sec 2023-03-31 18:54:53,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:53,321 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:53,322 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 18:54:53,322 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/C is initiating minor compaction (all files) 2023-03-31 18:54:53,322 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/C in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,322 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/389cb735f82a4ef7bc6e56c9586ffb4c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d7a8726ddc244f6dbdf5a4c01c6f0901, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/291a9634ca0d4914b19ee11ae56c442e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/112e2a8201cc4b92beaaa22006b3b781, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/43550e95942d42ab8f27c3b05f4bc669, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/29daf27829e345538860ce84f6ed5f11] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=69.6 K 2023-03-31 18:54:53,322 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 389cb735f82a4ef7bc6e56c9586ffb4c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680288888213 2023-03-31 18:54:53,322 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d7a8726ddc244f6dbdf5a4c01c6f0901, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680288888226 2023-03-31 18:54:53,323 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 291a9634ca0d4914b19ee11ae56c442e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680288888856 2023-03-31 18:54:53,323 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 112e2a8201cc4b92beaaa22006b3b781, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680288889478 2023-03-31 18:54:53,323 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 43550e95942d42ab8f27c3b05f4bc669, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680288890604 2023-03-31 18:54:53,324 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 29daf27829e345538860ce84f6ed5f11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680288891671 2023-03-31 18:54:53,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:53,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:53,325 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:53,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:53,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:53,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,327 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,328 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,328 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:53,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:53,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:53,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:53,329 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:53,329 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,329 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:53,340 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:53,340 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:53,350 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#C#compaction#197 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:53,350 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:53,351 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:54:53,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:53,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:53,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:53,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:53,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:53,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:53,390 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/d870bbdd2b6647d0aadb75ac0a62449a 2023-03-31 18:54:53,447 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/0ab380458a7c46a5b6b69e036c14f052 2023-03-31 18:54:53,459 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:53,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:53,507 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/3b5db18cac3e4ff78eeebcf4a854b197 2023-03-31 18:54:53,511 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/d870bbdd2b6647d0aadb75ac0a62449a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d870bbdd2b6647d0aadb75ac0a62449a 2023-03-31 18:54:53,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:53,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52530 deadline: 1680288953510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,515 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d870bbdd2b6647d0aadb75ac0a62449a, entries=150, sequenceid=112, filesize=11.6 K 2023-03-31 18:54:53,517 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/0ab380458a7c46a5b6b69e036c14f052 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ab380458a7c46a5b6b69e036c14f052 2023-03-31 18:54:53,521 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ab380458a7c46a5b6b69e036c14f052, entries=150, sequenceid=112, filesize=11.6 K 2023-03-31 18:54:53,522 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/3b5db18cac3e4ff78eeebcf4a854b197 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3b5db18cac3e4ff78eeebcf4a854b197 2023-03-31 18:54:53,528 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3b5db18cac3e4ff78eeebcf4a854b197, entries=150, sequenceid=112, filesize=11.6 K 2023-03-31 18:54:53,529 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2b9d344a6097fb5c7c41dc8319fac898 in 178ms, sequenceid=112, compaction requested=false 2023-03-31 18:54:53,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:53,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:53,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:53,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:53,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,532 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e80d8c2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:53,534 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:53,534 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,535 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:53,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:53,537 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:53,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:53,540 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:53,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:53,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:53,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:53,540 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:53,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,542 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,544 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:53,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:53,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:53,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,548 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:53,548 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:53,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:53,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288893493, max timestamp=1680288893615Num uniques 859; 2023-03-31 18:54:53,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:53,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288893493, max timestamp=1680288893615Num uniques 859; 2023-03-31 18:54:53,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:53,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288893493, max timestamp=1680288893615Num uniques 859; 2023-03-31 18:54:53,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:53,618 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=26.84 KB heapSize=66.23 KB 2023-03-31 18:54:53,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:53,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:53,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:53,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:53,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:53,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:53,648 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:53,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:53,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:53,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:53,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:53,652 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:53,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:53,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:53,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,656 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,657 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,657 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:53,658 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,658 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:53,658 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:53,658 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:53,659 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,659 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:53,659 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:53,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:53,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7adf7fbf[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:53,662 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:53,662 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,664 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:53,664 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:53,664 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:53,664 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,665 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:53,666 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:53,669 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:53,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:53,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:53,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:53,669 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:53,674 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,674 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,674 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,675 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,676 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,678 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,678 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,679 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:53,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,678 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,679 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,679 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,679 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,679 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:53,679 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,679 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:53,679 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,679 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288893674, max timestamp=1680288893685Num uniques 859; 2023-03-31 18:54:53,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:53,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288893674, max timestamp=1680288893685Num uniques 859; 2023-03-31 18:54:53,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:53,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288893674, max timestamp=1680288893685Num uniques 859; 2023-03-31 18:54:53,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:53,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288893731, max timestamp=1680288893750Num uniques 1018; 2023-03-31 18:54:53,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288893731, max timestamp=1680288893750Num uniques 1018; 2023-03-31 18:54:53,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288893731, max timestamp=1680288893750Num uniques 1018; 2023-03-31 18:54:53,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 18:54:53,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 18:54:53,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 18:54:53,775 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/cc3922d4b3e141c6a4d7b984709c2117 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/cc3922d4b3e141c6a4d7b984709c2117 2023-03-31 18:54:53,779 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:53,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:53,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:53,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:53,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:53,782 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:53,782 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,782 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:53,782 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:53,782 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,782 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,784 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,785 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:53,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:53,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:53,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:53,786 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,787 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:53,787 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/C of 2b9d344a6097fb5c7c41dc8319fac898 into cc3922d4b3e141c6a4d7b984709c2117(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:53,787 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:53,787 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/C, priority=10, startTime=1680288893222; duration=0sec 2023-03-31 18:54:53,787 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:53,787 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:53,787 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:53,789 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:53,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fcd7e81[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:53,791 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:53,791 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:53,794 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:53,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,795 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:53,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:53,800 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:53,800 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:53,800 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:53,800 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:53,800 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:53,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,805 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,806 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,808 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,810 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:53,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:53,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,811 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:53,811 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:53,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680288893806, max timestamp=1680288893823Num uniques 1027; 2023-03-31 18:54:53,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-31 18:54:53,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680288893806, max timestamp=1680288893823Num uniques 1027; 2023-03-31 18:54:53,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-31 18:54:53,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680288893806, max timestamp=1680288893823Num uniques 1027; 2023-03-31 18:54:53,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-31 18:54:53,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680288893860, max timestamp=1680288893874Num uniques 1036; 2023-03-31 18:54:53,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680288893860, max timestamp=1680288893874Num uniques 1036; 2023-03-31 18:54:53,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680288893860, max timestamp=1680288893874Num uniques 1036; 2023-03-31 18:54:53,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-31 18:54:53,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-31 18:54:53,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-31 18:54:53,911 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:53,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:53,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:53,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:53,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:53,918 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:53,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:53,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:53,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,919 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:53,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:53,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:53,928 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:53,928 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:53,928 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:53,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680288893907, max timestamp=1680288893928Num uniques 1045; 2023-03-31 18:54:53,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-31 18:54:53,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680288893907, max timestamp=1680288893928Num uniques 1045; 2023-03-31 18:54:53,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-31 18:54:53,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680288893907, max timestamp=1680288893928Num uniques 1045; 2023-03-31 18:54:53,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-31 18:54:53,940 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,940 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:53,940 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:53,940 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:53,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,942 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8a480d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:53,946 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:53,946 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,947 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,948 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,948 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:53,949 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:53,953 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:53,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:53,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:53,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:53,954 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:53,956 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,956 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:53,956 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:53,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,957 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,958 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,960 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,960 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,960 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:53,960 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:53,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:53,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:53,960 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,960 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:53,961 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:53,960 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:53,960 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:53,961 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:53,961 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:53,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:53,961 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:53,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:53,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680288893951, max timestamp=1680288893996Num uniques 1054; 2023-03-31 18:54:53,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-31 18:54:53,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680288893951, max timestamp=1680288893996Num uniques 1054; 2023-03-31 18:54:53,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-31 18:54:53,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680288893951, max timestamp=1680288893996Num uniques 1054; 2023-03-31 18:54:53,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-31 18:54:54,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/1314010fe6b74588874d44dbef07b0e4 2023-03-31 18:54:54,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680288894025, max timestamp=1680288894048Num uniques 1013; 2023-03-31 18:54:54,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-31 18:54:54,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680288894025, max timestamp=1680288894048Num uniques 1013; 2023-03-31 18:54:54,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680288894025, max timestamp=1680288894048Num uniques 1013; 2023-03-31 18:54:54,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-31 18:54:54,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-31 18:54:54,061 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:54,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:54,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:54,064 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:54,064 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,064 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:54,064 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:54,064 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,064 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,066 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,066 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,066 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:54,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:54,068 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,068 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:54,068 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:54,068 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,069 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:54,069 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:54,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42f4c956[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:54,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:54,077 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,078 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:54,078 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,079 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:54,080 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:54,080 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,080 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,080 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:54,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:54,082 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:54,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:54,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:54,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:54,082 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:54,083 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,084 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,085 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,088 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:54,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:54,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,089 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:54,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,089 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680288894087, max timestamp=1680288894112Num uniques 1072; 2023-03-31 18:54:54,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-31 18:54:54,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680288894087, max timestamp=1680288894112Num uniques 1072; 2023-03-31 18:54:54,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680288894087, max timestamp=1680288894112Num uniques 1072; 2023-03-31 18:54:54,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-31 18:54:54,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-31 18:54:54,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680288894138, max timestamp=1680288894162Num uniques 1031; 2023-03-31 18:54:54,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-31 18:54:54,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680288894138, max timestamp=1680288894162Num uniques 1031; 2023-03-31 18:54:54,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-31 18:54:54,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680288894138, max timestamp=1680288894162Num uniques 1031; 2023-03-31 18:54:54,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-31 18:54:54,189 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:54,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:54,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:54,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:54,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:54,192 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:54,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:54,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:54,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,194 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:54,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:54,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:54,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:54,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:54,196 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,196 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:54,196 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:54,196 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:54,198 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,198 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:54,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43d76430[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:54,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:54,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,201 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,201 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:54,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:54,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680288894153, max timestamp=1680288894201Num uniques 1040; 2023-03-31 18:54:54,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-31 18:54:54,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680288894153, max timestamp=1680288894201Num uniques 1040; 2023-03-31 18:54:54,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-31 18:54:54,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680288894153, max timestamp=1680288894201Num uniques 1040; 2023-03-31 18:54:54,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-31 18:54:54,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:54,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,204 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:54,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:54,207 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:54,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:54,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:54,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:54,207 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:54,208 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,208 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,208 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,209 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,210 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:54,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:54,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:54,212 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,213 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:54,213 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:54,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680288894253, max timestamp=1680288894265Num uniques 1099; 2023-03-31 18:54:54,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680288894253, max timestamp=1680288894265Num uniques 1099; 2023-03-31 18:54:54,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-03-31 18:54:54,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-03-31 18:54:54,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680288894253, max timestamp=1680288894265Num uniques 1099; 2023-03-31 18:54:54,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-03-31 18:54:54,313 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:54,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:54,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:54,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:54,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:54,316 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:54,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:54,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:54,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,318 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:54,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:54,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:54,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:54,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:54,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:54,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:54,320 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,320 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:54,320 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:54,320 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:54,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,322 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,324 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,324 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:54,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680288894313, max timestamp=1680288894323Num uniques 1058; 2023-03-31 18:54:54,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-31 18:54:54,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680288894313, max timestamp=1680288894323Num uniques 1058; 2023-03-31 18:54:54,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-31 18:54:54,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680288894313, max timestamp=1680288894323Num uniques 1058; 2023-03-31 18:54:54,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-31 18:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77da23bd[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:54,328 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:54,328 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:54,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:54,330 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,330 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,330 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,331 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:54,333 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:54,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:54,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:54,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:54,333 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:54,335 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,335 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,335 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:54,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:54,341 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:54,341 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:54,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,341 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680288894368, max timestamp=1680288894379Num uniques 1017; 2023-03-31 18:54:54,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680288894368, max timestamp=1680288894379Num uniques 1017; 2023-03-31 18:54:54,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-31 18:54:54,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-31 18:54:54,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680288894368, max timestamp=1680288894379Num uniques 1017; 2023-03-31 18:54:54,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-31 18:54:54,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680288894420, max timestamp=1680288894434Num uniques 1026; 2023-03-31 18:54:54,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-31 18:54:54,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680288894420, max timestamp=1680288894434Num uniques 1026; 2023-03-31 18:54:54,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680288894420, max timestamp=1680288894434Num uniques 1026; 2023-03-31 18:54:54,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-31 18:54:54,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-31 18:54:54,441 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:54,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:54,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:54,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:54,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:54,444 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:54,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:54,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:54,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:54,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:54,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:54,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:54,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:54,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:54,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:54,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:54,447 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,448 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:54,448 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:54,448 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:54,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:54,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:54,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f38c76b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:54,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,452 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:54,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:54,457 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/477457a6ffe942aab23f50934d6f2c09 2023-03-31 18:54:54,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:54,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,458 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,458 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:54,461 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:54,461 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:54,461 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:54,461 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:54,461 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:54,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,464 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,467 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,467 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/553454c5af034bf9bb48538a13fa6f96 2023-03-31 18:54:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:54,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:54,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,470 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,470 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:54,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,470 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:54,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,471 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/1314010fe6b74588874d44dbef07b0e4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1314010fe6b74588874d44dbef07b0e4 2023-03-31 18:54:54,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1314010fe6b74588874d44dbef07b0e4, entries=150, sequenceid=133, filesize=11.7 K 2023-03-31 18:54:54,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/477457a6ffe942aab23f50934d6f2c09 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/477457a6ffe942aab23f50934d6f2c09 2023-03-31 18:54:54,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/477457a6ffe942aab23f50934d6f2c09, entries=150, sequenceid=133, filesize=11.7 K 2023-03-31 18:54:54,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/553454c5af034bf9bb48538a13fa6f96 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/553454c5af034bf9bb48538a13fa6f96 2023-03-31 18:54:54,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/553454c5af034bf9bb48538a13fa6f96, entries=150, sequenceid=133, filesize=11.7 K 2023-03-31 18:54:54,485 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=127.47 KB/130530 for 2b9d344a6097fb5c7c41dc8319fac898 in 867ms, sequenceid=133, compaction requested=true 2023-03-31 18:54:54,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:54,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:54,485 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:54:54,485 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:54:54,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:54,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:54,486 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:54:54,486 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/A is initiating minor compaction (all files) 2023-03-31 18:54:54,486 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:54:54,486 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/A in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,486 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/B is initiating minor compaction (all files) 2023-03-31 18:54:54,486 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9fe90f870d7d4181976dea3fd84b7ba2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d870bbdd2b6647d0aadb75ac0a62449a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1314010fe6b74588874d44dbef07b0e4] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=35.1 K 2023-03-31 18:54:54,486 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/B in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,486 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/e97a507f776346e59e51e76ce313be00, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ab380458a7c46a5b6b69e036c14f052, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/477457a6ffe942aab23f50934d6f2c09] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=35.1 K 2023-03-31 18:54:54,487 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9fe90f870d7d4181976dea3fd84b7ba2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680288891671 2023-03-31 18:54:54,487 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e97a507f776346e59e51e76ce313be00, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680288891671 2023-03-31 18:54:54,487 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d870bbdd2b6647d0aadb75ac0a62449a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680288892312 2023-03-31 18:54:54,487 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 0ab380458a7c46a5b6b69e036c14f052, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680288892312 2023-03-31 18:54:54,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:54,488 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 477457a6ffe942aab23f50934d6f2c09, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680288893493 2023-03-31 18:54:54,488 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 1314010fe6b74588874d44dbef07b0e4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680288893493 2023-03-31 18:54:54,488 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=134.18 KB heapSize=383.39 KB 2023-03-31 18:54:54,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:54,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:54,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:54,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:54,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:54,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:54,506 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#A#compaction#205 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:54,509 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#B#compaction#206 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:54,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:54,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288954519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/8af81cd69cbf48f88659c5895069baef 2023-03-31 18:54:54,571 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:54,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:54,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:54,576 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/46fcedcb3e99493395a0caa2d2567ccc as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/46fcedcb3e99493395a0caa2d2567ccc 2023-03-31 18:54:54,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:54,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:54,579 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:54,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:54,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:54,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,581 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,581 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:54,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:54,584 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/A of 2b9d344a6097fb5c7c41dc8319fac898 into 46fcedcb3e99493395a0caa2d2567ccc(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:54,584 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/224980c1da734dfa928be8ca0af992f8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/224980c1da734dfa928be8ca0af992f8 2023-03-31 18:54:54,584 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:54,585 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/A, priority=13, startTime=1680288894485; duration=0sec 2023-03-31 18:54:54,585 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:54,585 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:54:54,586 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:54:54,587 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/C is initiating minor compaction (all files) 2023-03-31 18:54:54,587 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/C in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,587 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/cc3922d4b3e141c6a4d7b984709c2117, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3b5db18cac3e4ff78eeebcf4a854b197, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/553454c5af034bf9bb48538a13fa6f96] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=35.1 K 2023-03-31 18:54:54,588 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting cc3922d4b3e141c6a4d7b984709c2117, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680288891671 2023-03-31 18:54:54,588 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 3b5db18cac3e4ff78eeebcf4a854b197, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680288892312 2023-03-31 18:54:54,589 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 553454c5af034bf9bb48538a13fa6f96, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680288893493 2023-03-31 18:54:54,591 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/B of 2b9d344a6097fb5c7c41dc8319fac898 into 224980c1da734dfa928be8ca0af992f8(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:54,591 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:54,591 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/B, priority=13, startTime=1680288894485; duration=0sec 2023-03-31 18:54:54,591 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:54,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:54,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:54,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:54,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:54,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:54,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:54,594 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,594 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:54,594 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:54,594 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:54,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:54,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:54,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:54,606 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,606 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,607 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,607 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:54,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b620ba3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:54,609 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:54,609 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:54,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:54,612 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:54,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:54,615 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:54,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:54,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:54,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:54,615 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:54,616 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,616 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,616 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,620 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,624 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:54,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:54,624 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:54,624 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:54,624 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,624 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,624 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,624 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,624 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,624 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,624 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:54,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:54,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288954625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,633 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#C#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:54,724 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:54,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:54,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:54,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:54,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:54,732 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:54,732 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,732 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:54,732 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:54,732 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,732 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,734 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,734 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,734 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,734 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:54,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:54,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:54,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288954735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:54,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288954747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:54,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:54,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:54,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:54,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:54,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,761 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:54,761 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:54,761 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:54,760 DEBUG [Thread-1521] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:54,760 DEBUG [Thread-1525] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:54,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:54,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288954761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,763 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:54,765 DEBUG [Thread-1523] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288954763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,768 DEBUG [Thread-1527] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@436c0e1b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:54,770 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:54,770 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,771 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:54,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:54,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:54,779 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:54,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:54,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:54,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:54,779 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:54,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,783 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:54,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:54,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,787 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,787 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:54,788 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:54,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:54,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288954829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,888 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:54,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:54,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:54,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:54,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:54,891 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:54,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:54,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:54,892 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,892 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,893 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:54,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:54,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:54,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:54,895 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:54,895 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:54,895 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,895 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:54,895 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:54,895 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:54,895 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:54,895 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:54,895 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,895 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:54,895 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:54,897 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,897 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:54,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650d0a00[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:54,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:54,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:54,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,901 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:54,904 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:54,904 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:54,904 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:54,904 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:54,904 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:54,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:54,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,910 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:54,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,910 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,911 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,913 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,913 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:54,913 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:54,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:54,913 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:54,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:54,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:54,913 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:54,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:54,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:54,914 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:54,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,914 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:54,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:54,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,013 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/19ebf44552ed4de2afdbbd28dd73ccb5 2023-03-31 18:54:55,014 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:55,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:55,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:55,018 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:55,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:55,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:55,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,020 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,021 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,021 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:55,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:55,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:55,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:55,026 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:55,026 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,026 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:55,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:55,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:55,028 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,028 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:55,028 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:55,028 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:55,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:55,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:55,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:55,030 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,030 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:55,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:55,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c978a55[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:55,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:55,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,039 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:55,039 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,039 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:55,041 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:55,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:55,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:55,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:55,044 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:55,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:55,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:55,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:55,044 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:55,046 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,046 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,046 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:55,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,057 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/c2211646d9414331b46ec4d6a406967f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c2211646d9414331b46ec4d6a406967f 2023-03-31 18:54:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,063 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/C of 2b9d344a6097fb5c7c41dc8319fac898 into c2211646d9414331b46ec4d6a406967f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:55,063 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:55,063 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/C, priority=13, startTime=1680288894485; duration=0sec 2023-03-31 18:54:55,063 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:55,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:55,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,065 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,065 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:55,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:55,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:55,065 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:55,065 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:55,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:55,065 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,071 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/3f127b6976cf4942aa8d68ac0e5468b2 2023-03-31 18:54:55,076 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/8af81cd69cbf48f88659c5895069baef as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/8af81cd69cbf48f88659c5895069baef 2023-03-31 18:54:55,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/8af81cd69cbf48f88659c5895069baef, entries=200, sequenceid=394, filesize=14.3 K 2023-03-31 18:54:55,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/19ebf44552ed4de2afdbbd28dd73ccb5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19ebf44552ed4de2afdbbd28dd73ccb5 2023-03-31 18:54:55,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19ebf44552ed4de2afdbbd28dd73ccb5, entries=150, sequenceid=394, filesize=11.9 K 2023-03-31 18:54:55,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/3f127b6976cf4942aa8d68ac0e5468b2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3f127b6976cf4942aa8d68ac0e5468b2 2023-03-31 18:54:55,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3f127b6976cf4942aa8d68ac0e5468b2, entries=150, sequenceid=394, filesize=11.9 K 2023-03-31 18:54:55,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~383.34 KB/392544, currentSize=46.96 KB/48090 for 2b9d344a6097fb5c7c41dc8319fac898 in 612ms, sequenceid=394, compaction requested=false 2023-03-31 18:54:55,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:55,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:55,136 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:54:55,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:55,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:55,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:55,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:55,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:55,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:55,148 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/fe0b653a52484e4aba576154b297e44f 2023-03-31 18:54:55,165 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:55,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:55,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:55,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:55,169 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:55,169 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,169 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:55,169 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:55,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:55,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/10d5a4133d3b4ba1ab051e6a0070d364 2023-03-31 18:54:55,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:55,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:55,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:55,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:55,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:55,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:55,174 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,174 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:55,175 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:55,175 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:55,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:55,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:55,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:55,176 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:55,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5703a977[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:55,179 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:55,179 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:55,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:55,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:55,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:55,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:55,185 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:55,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:55,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:55,186 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:55,186 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:55,191 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,191 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,191 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:55,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,196 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:55,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:55,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:55,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:55,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,197 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:55,197 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:55,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:55,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288895186, max timestamp=1680288895201Num uniques 859; 2023-03-31 18:54:55,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:55,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288895186, max timestamp=1680288895201Num uniques 859; 2023-03-31 18:54:55,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288895186, max timestamp=1680288895201Num uniques 859; 2023-03-31 18:54:55,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:55,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:55,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:55,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288955247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,297 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:55,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:55,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:55,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:55,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:55,300 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:55,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:55,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:55,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,301 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:55,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:55,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:55,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:55,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:55,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:55,303 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,303 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:55,304 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:55,304 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:55,305 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,305 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,305 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,305 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:55,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@743c795a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:55,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,309 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,309 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:55,309 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:55,309 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,309 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:55,309 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:55,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:55,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:55,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:55,312 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:55,312 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:55,312 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:55,312 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:55,312 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:55,314 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,314 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,314 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:55,314 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,321 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,327 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:55,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:55,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:55,328 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,328 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:55,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,328 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:55,328 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:55,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:55,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:55,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288955352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,428 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:55,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:55,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:55,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:55,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:55,432 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:55,432 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,432 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:55,432 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:55,432 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,432 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,434 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,434 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,434 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,434 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:55,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:55,435 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,435 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:55,436 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:55,436 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:55,437 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:55,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:55,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ec30d7[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:55,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,440 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:55,440 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,440 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:55,443 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:55,445 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:55,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:55,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:55,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:55,445 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:55,447 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,447 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,447 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,447 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,449 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:55,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:55,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:55,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:55,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:55,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,452 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,452 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,452 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:55,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:55,452 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:55,452 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,452 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,552 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:55,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:55,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:55,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:55,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:55,555 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:55,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:55,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:55,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:55,558 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288955556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,558 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:55,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:55,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:55,560 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,560 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:55,560 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:55,560 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:55,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39e9d89d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:55,564 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:55,564 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,565 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:55,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:55,566 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:55,569 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:55,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:55,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:55,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:55,569 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:55,570 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,570 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:55,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,570 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:55,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,571 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,572 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,577 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,577 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,577 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:55,577 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:55,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:55,577 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:55,577 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,578 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,578 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:55,578 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,578 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:55,578 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:55,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:55,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/d143781f1367497794737b01d88efb67 2023-03-31 18:54:55,591 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/fe0b653a52484e4aba576154b297e44f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fe0b653a52484e4aba576154b297e44f 2023-03-31 18:54:55,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fe0b653a52484e4aba576154b297e44f, entries=150, sequenceid=408, filesize=11.9 K 2023-03-31 18:54:55,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/10d5a4133d3b4ba1ab051e6a0070d364 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10d5a4133d3b4ba1ab051e6a0070d364 2023-03-31 18:54:55,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10d5a4133d3b4ba1ab051e6a0070d364, entries=100, sequenceid=408, filesize=9.5 K 2023-03-31 18:54:55,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/d143781f1367497794737b01d88efb67 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d143781f1367497794737b01d88efb67 2023-03-31 18:54:55,606 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d143781f1367497794737b01d88efb67, entries=100, sequenceid=408, filesize=9.5 K 2023-03-31 18:54:55,607 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2b9d344a6097fb5c7c41dc8319fac898 in 471ms, sequenceid=408, compaction requested=true 2023-03-31 18:54:55,607 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:55,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:55,607 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:54:55,607 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:54:55,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:55,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:55,608 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:54:55,609 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/A is initiating minor compaction (all files) 2023-03-31 18:54:55,609 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/A in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,609 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/46fcedcb3e99493395a0caa2d2567ccc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/8af81cd69cbf48f88659c5895069baef, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fe0b653a52484e4aba576154b297e44f] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=38.2 K 2023-03-31 18:54:55,609 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34222 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:54:55,609 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/B is initiating minor compaction (all files) 2023-03-31 18:54:55,609 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/B in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,609 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/224980c1da734dfa928be8ca0af992f8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19ebf44552ed4de2afdbbd28dd73ccb5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10d5a4133d3b4ba1ab051e6a0070d364] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=33.4 K 2023-03-31 18:54:55,609 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 46fcedcb3e99493395a0caa2d2567ccc, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680288893493 2023-03-31 18:54:55,610 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 224980c1da734dfa928be8ca0af992f8, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680288893493 2023-03-31 18:54:55,610 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8af81cd69cbf48f88659c5895069baef, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680288894474 2023-03-31 18:54:55,610 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 19ebf44552ed4de2afdbbd28dd73ccb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680288894474 2023-03-31 18:54:55,610 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting fe0b653a52484e4aba576154b297e44f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288894511 2023-03-31 18:54:55,610 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 10d5a4133d3b4ba1ab051e6a0070d364, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288894517 2023-03-31 18:54:55,617 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#B#compaction#213 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:55,618 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#A#compaction#214 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:55,678 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:55,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:55,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:55,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:55,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:55,681 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:55,681 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,681 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:55,681 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:55,682 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,682 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,683 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,683 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:55,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:55,684 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,684 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:55,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:55,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:55,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:55,686 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:55,686 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:55,686 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-31 18:54:55,687 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:55,687 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:55,687 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:55,687 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:55,687 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:55,687 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:55,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:55,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:55,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:55,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288955885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:55,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:55,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288955989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,029 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/d55ce6da052d4854973daa67ed20bc52 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/d55ce6da052d4854973daa67ed20bc52 2023-03-31 18:54:56,031 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/f181863dd157479da725714d7f494640 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/f181863dd157479da725714d7f494640 2023-03-31 18:54:56,045 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/A of 2b9d344a6097fb5c7c41dc8319fac898 into f181863dd157479da725714d7f494640(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:56,045 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/B of 2b9d344a6097fb5c7c41dc8319fac898 into d55ce6da052d4854973daa67ed20bc52(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:56,045 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:56,045 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:56,045 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/A, priority=13, startTime=1680288895607; duration=0sec 2023-03-31 18:54:56,045 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/B, priority=13, startTime=1680288895607; duration=0sec 2023-03-31 18:54:56,045 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:56,045 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:54:56,045 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:54:56,046 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34222 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:54:56,047 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/C is initiating minor compaction (all files) 2023-03-31 18:54:56,047 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/C in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:56,047 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c2211646d9414331b46ec4d6a406967f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3f127b6976cf4942aa8d68ac0e5468b2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d143781f1367497794737b01d88efb67] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=33.4 K 2023-03-31 18:54:56,047 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c2211646d9414331b46ec4d6a406967f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680288893493 2023-03-31 18:54:56,047 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 3f127b6976cf4942aa8d68ac0e5468b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680288894474 2023-03-31 18:54:56,048 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d143781f1367497794737b01d88efb67, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288894517 2023-03-31 18:54:56,053 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#C#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:54:56,095 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/791e62679b6a4fba82d2a2dc2ec5f01e 2023-03-31 18:54:56,106 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/150b88c9c0c44e19ba96e18928565a31 2023-03-31 18:54:56,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:56,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288956192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,463 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/e41912e44c214deeaa3caa026f95f91b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/e41912e44c214deeaa3caa026f95f91b 2023-03-31 18:54:56,467 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/C of 2b9d344a6097fb5c7c41dc8319fac898 into e41912e44c214deeaa3caa026f95f91b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:54:56,467 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:56,467 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/C, priority=13, startTime=1680288895607; duration=0sec 2023-03-31 18:54:56,467 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:54:56,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:56,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288956497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,531 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/bbf9cd8811584db1933950b41f2a08b7 2023-03-31 18:54:56,535 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/791e62679b6a4fba82d2a2dc2ec5f01e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/791e62679b6a4fba82d2a2dc2ec5f01e 2023-03-31 18:54:56,540 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/791e62679b6a4fba82d2a2dc2ec5f01e, entries=150, sequenceid=444, filesize=11.9 K 2023-03-31 18:54:56,541 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/150b88c9c0c44e19ba96e18928565a31 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/150b88c9c0c44e19ba96e18928565a31 2023-03-31 18:54:56,544 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/150b88c9c0c44e19ba96e18928565a31, entries=150, sequenceid=444, filesize=11.9 K 2023-03-31 18:54:56,545 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/bbf9cd8811584db1933950b41f2a08b7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/bbf9cd8811584db1933950b41f2a08b7 2023-03-31 18:54:56,549 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/bbf9cd8811584db1933950b41f2a08b7, entries=150, sequenceid=444, filesize=11.9 K 2023-03-31 18:54:56,550 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 2b9d344a6097fb5c7c41dc8319fac898 in 864ms, sequenceid=444, compaction requested=false 2023-03-31 18:54:56,550 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:56,550 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:56,550 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:56,550 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:56,550 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,550 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:56,550 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:56,551 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:56,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:56,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736b96c6[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:56,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:56,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:56,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:56,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:56,555 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:56,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:56,555 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:56,555 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:56,557 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:56,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:56,560 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:56,560 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:56,560 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:56,560 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:56,560 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:56,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:56,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:56,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:56,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:56,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:56,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:56,564 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:56,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:56,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:56,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:56,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:56,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:56,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:56,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:56,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:56,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:56,567 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:56,567 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:56,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:56,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:56,567 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:56,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:56,667 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:56,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:56,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:56,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:56,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:56,671 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:56,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:56,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:56,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:56,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:56,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:56,679 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,679 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:56,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:56,680 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:56,680 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:56,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:56,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:56,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:56,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:56,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:56,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:56,681 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:54:56,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:56,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:56,682 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:56,682 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:56,682 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:56,682 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:56,690 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/599bccb5417f4f848f2d4074aba64901 2023-03-31 18:54:57,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:57,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288897039, max timestamp=1680288897053Num uniques 859; 2023-03-31 18:54:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288897039, max timestamp=1680288897053Num uniques 859; 2023-03-31 18:54:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288897039, max timestamp=1680288897053Num uniques 859; 2023-03-31 18:54:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:54:57,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:57,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288957099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,119 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/2cb0c2205c8141829bd4e9e9cf94caf5 2023-03-31 18:54:57,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:57,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 454 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288957202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:57,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288957406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,546 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/f4b1fb97c625488f8d97463d352c89b1 2023-03-31 18:54:57,550 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/599bccb5417f4f848f2d4074aba64901 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/599bccb5417f4f848f2d4074aba64901 2023-03-31 18:54:57,554 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/599bccb5417f4f848f2d4074aba64901, entries=150, sequenceid=458, filesize=11.9 K 2023-03-31 18:54:57,555 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/2cb0c2205c8141829bd4e9e9cf94caf5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2cb0c2205c8141829bd4e9e9cf94caf5 2023-03-31 18:54:57,559 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2cb0c2205c8141829bd4e9e9cf94caf5, entries=150, sequenceid=458, filesize=11.9 K 2023-03-31 18:54:57,559 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/f4b1fb97c625488f8d97463d352c89b1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/f4b1fb97c625488f8d97463d352c89b1 2023-03-31 18:54:57,569 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/f4b1fb97c625488f8d97463d352c89b1, entries=150, sequenceid=458, filesize=11.9 K 2023-03-31 18:54:57,570 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2b9d344a6097fb5c7c41dc8319fac898 in 889ms, sequenceid=458, compaction requested=true 2023-03-31 18:54:57,570 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:57,570 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:57,571 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@648ac758[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:57,573 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:57,573 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:57,574 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:57,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:57,574 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,574 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:57,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:57,576 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:57,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:57,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:57,578 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:57,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:57,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:57,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:57,578 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:57,581 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:57,581 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:57,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:57,581 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:57,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:57,581 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:57,583 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:57,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:57,585 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:57,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:57,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:57,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:57,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:57,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:57,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:57,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:57,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:57,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:57,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:57,586 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:57,586 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:57,686 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:57,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:57,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:57,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:57,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:57,690 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:57,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:57,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:57,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:57,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:57,690 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:57,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:57,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:57,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:57,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:57,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:57,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:57,694 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:57,694 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:57,694 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:57,694 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:57,694 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-31 18:54:57,694 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:57,694 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:57,694 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:57,694 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:57,694 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:57,694 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:57,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:57,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:57,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 466 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288957749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:57,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:57,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 468 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288957852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 470 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288958056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,102 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/914612b56b35466793ae61dbe64984d2 2023-03-31 18:54:58,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288958359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,516 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/19edb54ba7894a8e8185746d94092aa8 2023-03-31 18:54:58,528 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/c50e5610640b49a18fdec1f003e177e4 2023-03-31 18:54:58,532 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/914612b56b35466793ae61dbe64984d2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/914612b56b35466793ae61dbe64984d2 2023-03-31 18:54:58,536 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/914612b56b35466793ae61dbe64984d2, entries=150, sequenceid=494, filesize=11.9 K 2023-03-31 18:54:58,537 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/19edb54ba7894a8e8185746d94092aa8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19edb54ba7894a8e8185746d94092aa8 2023-03-31 18:54:58,541 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19edb54ba7894a8e8185746d94092aa8, entries=150, sequenceid=494, filesize=11.9 K 2023-03-31 18:54:58,542 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/c50e5610640b49a18fdec1f003e177e4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c50e5610640b49a18fdec1f003e177e4 2023-03-31 18:54:58,545 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c50e5610640b49a18fdec1f003e177e4, entries=150, sequenceid=494, filesize=11.9 K 2023-03-31 18:54:58,546 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 2b9d344a6097fb5c7c41dc8319fac898 in 852ms, sequenceid=494, compaction requested=true 2023-03-31 18:54:58,546 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:58,546 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:58,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:58,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:58,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf82996[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:58,550 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:58,550 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:58,552 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:58,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:58,552 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,552 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:58,553 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,553 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:58,553 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:58,553 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:58,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:58,557 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:58,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:58,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:58,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:58,557 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:58,558 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:58,559 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:58,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:58,559 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:58,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:58,561 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,564 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:58,564 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:58,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:58,564 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:58,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:58,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:58,565 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:58,565 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:58,565 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,565 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:58,565 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:58,565 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:58,565 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:58,565 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:58,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:58,665 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:58,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:58,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:58,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:58,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:58,669 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:58,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:58,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:58,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:58,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:58,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:58,671 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:58,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:58,672 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,672 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:58,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:58,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:58,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:58,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:58,673 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:58,673 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:58,673 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:58,673 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:58,673 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:58,673 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:58,673 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:58,673 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:54:58,674 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:58,674 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:58,674 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:58,674 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:58,674 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:58,674 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:54:58,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:58,765 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:58,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288958794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288898792, max timestamp=1680288898796Num uniques 859; 2023-03-31 18:54:58,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:54:58,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288898792, max timestamp=1680288898796Num uniques 859; 2023-03-31 18:54:58,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288898792, max timestamp=1680288898796Num uniques 859; 2023-03-31 18:54:58,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:54:58,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:54:58,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288958813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288958813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288958816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 474 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288958866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288958898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288958917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288958917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:58,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:58,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288958920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,086 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/e13806f21d6b4f62b01212214f3afef9 2023-03-31 18:54:59,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288959102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288959122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288959122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288959124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,130 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/de7b6a17d29e40ddb36709043e5057b9 2023-03-31 18:54:59,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288959406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288959426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288959427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288959429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,548 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/421161df695340788772c14c10d5f67a 2023-03-31 18:54:59,553 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/e13806f21d6b4f62b01212214f3afef9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e13806f21d6b4f62b01212214f3afef9 2023-03-31 18:54:59,557 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e13806f21d6b4f62b01212214f3afef9, entries=150, sequenceid=505, filesize=11.9 K 2023-03-31 18:54:59,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/de7b6a17d29e40ddb36709043e5057b9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/de7b6a17d29e40ddb36709043e5057b9 2023-03-31 18:54:59,563 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/de7b6a17d29e40ddb36709043e5057b9, entries=150, sequenceid=505, filesize=11.9 K 2023-03-31 18:54:59,564 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/421161df695340788772c14c10d5f67a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/421161df695340788772c14c10d5f67a 2023-03-31 18:54:59,568 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/421161df695340788772c14c10d5f67a, entries=150, sequenceid=505, filesize=11.9 K 2023-03-31 18:54:59,569 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2b9d344a6097fb5c7c41dc8319fac898 in 896ms, sequenceid=505, compaction requested=true 2023-03-31 18:54:59,569 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:54:59,569 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:59,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:54:59,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:54:59,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:54:59,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:54:59,571 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:59,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:59,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:54:59,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:59,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:59,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:54:59,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b223ba4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:54:59,573 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:54:59,573 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:59,575 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:59,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:59,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:54:59,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:54:59,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:54:59,575 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,575 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:54:59,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:54:59,576 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:59,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:54:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:59,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:59,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:54:59,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:54:59,580 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:54:59,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:54:59,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:54:59,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:54:59,580 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:54:59,583 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:59,583 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:59,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:59,583 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:54:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:54:59,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:54:59,584 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:54:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:59,585 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:54:59,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:54:59,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:59,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:54:59,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:54:59,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:54:59,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:59,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:54:59,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:59,587 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:54:59,587 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:54:59,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:59,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,587 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:54:59,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:59,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:54:59,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:54:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:54:59,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:59,687 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:54:59,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:54:59,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:54:59,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:54:59,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:54:59,695 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:54:59,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:59,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:54:59,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:54:59,696 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:59,696 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:59,698 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,698 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:54:59,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:54:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:54:59,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:59,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:54:59,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:54:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:54:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:54:59,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:54:59,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:54:59,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:54:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:54:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:54:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:54:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:54:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:54:59,702 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:54:59,702 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:54:59,702 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-31 18:54:59,702 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:54:59,702 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:59,702 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:54:59,702 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:59,703 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:54:59,703 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:54:59,738 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/e21c95bcf0ae4dd18e780274c3490e35 2023-03-31 18:54:59,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:54:59,878 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:54:59,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288959903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288959910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288959931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288959932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:54:59,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:54:59,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288959936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:00,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288960010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,153 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/698a26e16cd84077b54fdd6933264995 2023-03-31 18:55:00,165 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/add6769a58ea4b0983a2f94d888f0a13 2023-03-31 18:55:00,173 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/e21c95bcf0ae4dd18e780274c3490e35 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e21c95bcf0ae4dd18e780274c3490e35 2023-03-31 18:55:00,177 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e21c95bcf0ae4dd18e780274c3490e35, entries=150, sequenceid=540, filesize=11.9 K 2023-03-31 18:55:00,178 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/698a26e16cd84077b54fdd6933264995 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/698a26e16cd84077b54fdd6933264995 2023-03-31 18:55:00,181 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/698a26e16cd84077b54fdd6933264995, entries=150, sequenceid=540, filesize=11.9 K 2023-03-31 18:55:00,182 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/add6769a58ea4b0983a2f94d888f0a13 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/add6769a58ea4b0983a2f94d888f0a13 2023-03-31 18:55:00,194 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/add6769a58ea4b0983a2f94d888f0a13, entries=150, sequenceid=540, filesize=11.9 K 2023-03-31 18:55:00,195 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 2b9d344a6097fb5c7c41dc8319fac898 in 493ms, sequenceid=540, compaction requested=true 2023-03-31 18:55:00,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:00,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:00,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:00,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:00,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:00,203 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,203 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,203 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,203 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:00,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24d051a0[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:00,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:00,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:00,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:00,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:00,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:00,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:00,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:00,208 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:00,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:00,210 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:00,210 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:00,210 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:00,210 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:00,210 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:00,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:00,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:00,216 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:55:00,216 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:00,216 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:00,216 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:00,216 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:00,216 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:00,216 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:00,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:00,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:00,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:00,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:00,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:00,219 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,219 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:00,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:00,219 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:00,219 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:00,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/74f2029adf834ec7bfc357623c98d86f 2023-03-31 18:55:00,243 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/048be2d608434ed4a43fe67b728b446c 2023-03-31 18:55:00,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:00,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 506 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288960296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,319 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:00,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:00,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:00,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:00,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:00,322 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:00,322 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:00,322 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:00,322 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:00,323 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,323 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,324 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:00,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:00,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:00,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:00,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:00,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:00,325 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:00,325 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:00,325 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:00,325 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:00,326 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,326 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:00,326 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:00,326 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:00,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,328 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:00,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1652f265[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:00,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:00,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,330 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,330 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:00,330 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:00,330 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:00,330 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,330 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:00,333 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:00,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:00,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:00,335 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:00,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:00,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:00,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:00,335 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:00,336 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,336 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:00,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,336 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:00,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,338 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:00,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:00,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:00,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:00,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:00,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:00,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:00,340 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:00,341 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:00,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:00,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 508 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288960401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,441 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:00,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:00,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:00,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:00,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:00,444 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:00,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:00,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:00,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:00,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:00,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:00,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:00,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:00,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:00,447 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,447 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:00,447 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:00,447 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:00,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,451 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,451 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:00,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b5bcda0[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:00,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:00,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:00,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:00,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:00,454 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,454 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:00,454 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:00,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:00,457 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:00,457 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:00,457 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:00,457 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:00,457 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:00,458 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,458 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,458 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:00,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,460 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:00,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:00,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:00,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:00,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:00,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:00,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:00,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:00,464 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:00,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,464 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:00,564 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:00,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:00,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:00,567 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:00,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:00,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:00,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:00,568 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,568 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:00,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:00,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:00,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:00,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:00,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:00,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:00,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:00,576 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,576 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:00,577 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:00,577 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:00,578 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:00,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@313e2c74[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:00,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:00,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,583 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:00,583 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:00,583 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:00,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:00,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:00,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:00,592 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:00,592 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:00,592 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:00,592 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:00,592 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:00,593 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:00,594 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,595 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:00,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,598 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:00,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:00,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:00,599 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:00,599 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:00,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:00,600 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,600 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:00,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,600 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:00,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:00,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:00,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:00,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:00,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 510 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288960607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/4534ce77dc3049f796354df01f13a8e0 2023-03-31 18:55:00,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/74f2029adf834ec7bfc357623c98d86f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/74f2029adf834ec7bfc357623c98d86f 2023-03-31 18:55:00,660 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/74f2029adf834ec7bfc357623c98d86f, entries=200, sequenceid=552, filesize=14.3 K 2023-03-31 18:55:00,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/048be2d608434ed4a43fe67b728b446c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/048be2d608434ed4a43fe67b728b446c 2023-03-31 18:55:00,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/048be2d608434ed4a43fe67b728b446c, entries=150, sequenceid=552, filesize=11.9 K 2023-03-31 18:55:00,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/4534ce77dc3049f796354df01f13a8e0 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4534ce77dc3049f796354df01f13a8e0 2023-03-31 18:55:00,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4534ce77dc3049f796354df01f13a8e0, entries=150, sequenceid=552, filesize=11.9 K 2023-03-31 18:55:00,669 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2b9d344a6097fb5c7c41dc8319fac898 in 453ms, sequenceid=552, compaction requested=true 2023-03-31 18:55:00,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:00,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:00,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:00,669 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-31 18:55:00,669 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-31 18:55:00,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:00,671 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85712 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-31 18:55:00,672 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/B is initiating minor compaction (all files) 2023-03-31 18:55:00,672 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/B in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,672 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88152 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-31 18:55:00,672 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/d55ce6da052d4854973daa67ed20bc52, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/150b88c9c0c44e19ba96e18928565a31, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2cb0c2205c8141829bd4e9e9cf94caf5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19edb54ba7894a8e8185746d94092aa8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/de7b6a17d29e40ddb36709043e5057b9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/698a26e16cd84077b54fdd6933264995, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/048be2d608434ed4a43fe67b728b446c] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=83.7 K 2023-03-31 18:55:00,672 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/A is initiating minor compaction (all files) 2023-03-31 18:55:00,672 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/A in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,672 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/f181863dd157479da725714d7f494640, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/791e62679b6a4fba82d2a2dc2ec5f01e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/599bccb5417f4f848f2d4074aba64901, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/914612b56b35466793ae61dbe64984d2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e13806f21d6b4f62b01212214f3afef9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e21c95bcf0ae4dd18e780274c3490e35, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/74f2029adf834ec7bfc357623c98d86f] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=86.1 K 2023-03-31 18:55:00,672 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d55ce6da052d4854973daa67ed20bc52, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288894487 2023-03-31 18:55:00,672 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting f181863dd157479da725714d7f494640, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288894487 2023-03-31 18:55:00,673 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 150b88c9c0c44e19ba96e18928565a31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680288895228 2023-03-31 18:55:00,673 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 791e62679b6a4fba82d2a2dc2ec5f01e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680288895228 2023-03-31 18:55:00,673 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2cb0c2205c8141829bd4e9e9cf94caf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680288895877 2023-03-31 18:55:00,673 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 599bccb5417f4f848f2d4074aba64901, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680288895877 2023-03-31 18:55:00,677 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 19edb54ba7894a8e8185746d94092aa8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680288897091 2023-03-31 18:55:00,677 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 914612b56b35466793ae61dbe64984d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680288897091 2023-03-31 18:55:00,684 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting de7b6a17d29e40ddb36709043e5057b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680288897725 2023-03-31 18:55:00,684 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting e13806f21d6b4f62b01212214f3afef9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680288897725 2023-03-31 18:55:00,685 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 698a26e16cd84077b54fdd6933264995, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680288898809 2023-03-31 18:55:00,685 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting e21c95bcf0ae4dd18e780274c3490e35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680288898809 2023-03-31 18:55:00,685 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 048be2d608434ed4a43fe67b728b446c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680288899877 2023-03-31 18:55:00,686 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 74f2029adf834ec7bfc357623c98d86f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680288899877 2023-03-31 18:55:00,699 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:00,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:00,708 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#B#compaction#234 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:00,709 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#A#compaction#235 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:00,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:00,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:00,710 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:00,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:00,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:00,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:00,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,713 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:00,713 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:00,713 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,713 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:00,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:00,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:00,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:00,727 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:00,727 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:00,727 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,727 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:00,727 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:55:00,728 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:00,728 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:00,728 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:00,728 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:00,728 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:00,728 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:00,735 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/fc41c3582a7143f0911808d57c24aa6d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fc41c3582a7143f0911808d57c24aa6d 2023-03-31 18:55:00,739 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/A of 2b9d344a6097fb5c7c41dc8319fac898 into fc41c3582a7143f0911808d57c24aa6d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:00,739 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:00,739 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/A, priority=9, startTime=1680288900669; duration=0sec 2023-03-31 18:55:00,740 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:00,740 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-31 18:55:00,742 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85712 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-31 18:55:00,742 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/C is initiating minor compaction (all files) 2023-03-31 18:55:00,742 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/C in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:00,743 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/e41912e44c214deeaa3caa026f95f91b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/bbf9cd8811584db1933950b41f2a08b7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/f4b1fb97c625488f8d97463d352c89b1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c50e5610640b49a18fdec1f003e177e4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/421161df695340788772c14c10d5f67a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/add6769a58ea4b0983a2f94d888f0a13, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4534ce77dc3049f796354df01f13a8e0] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=83.7 K 2023-03-31 18:55:00,743 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting e41912e44c214deeaa3caa026f95f91b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680288894487 2023-03-31 18:55:00,743 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting bbf9cd8811584db1933950b41f2a08b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680288895228 2023-03-31 18:55:00,743 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting f4b1fb97c625488f8d97463d352c89b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680288895877 2023-03-31 18:55:00,744 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c50e5610640b49a18fdec1f003e177e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680288897091 2023-03-31 18:55:00,744 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 421161df695340788772c14c10d5f67a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680288897725 2023-03-31 18:55:00,744 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting add6769a58ea4b0983a2f94d888f0a13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680288898809 2023-03-31 18:55:00,745 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 4534ce77dc3049f796354df01f13a8e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680288899877 2023-03-31 18:55:00,758 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#C#compaction#237 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:00,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:00,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:00,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:00,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288960928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:00,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 518 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288960930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:00,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288960936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:00,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288960941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:00,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:00,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288960944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:01,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288961031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:01,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 520 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288961033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,126 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/db1e884a99c64f458a4ebf18a6ad7854 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/db1e884a99c64f458a4ebf18a6ad7854 2023-03-31 18:55:01,130 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/B of 2b9d344a6097fb5c7c41dc8319fac898 into db1e884a99c64f458a4ebf18a6ad7854(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:01,131 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:01,131 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/B, priority=9, startTime=1680288900669; duration=0sec 2023-03-31 18:55:01,131 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:01,142 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/e4d3ace0400449599cfef3d0b618d5d8 2023-03-31 18:55:01,177 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/510a4dbe791c4b3c836172ce4334eb01 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/510a4dbe791c4b3c836172ce4334eb01 2023-03-31 18:55:01,181 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/C of 2b9d344a6097fb5c7c41dc8319fac898 into 510a4dbe791c4b3c836172ce4334eb01(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:01,181 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:01,181 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/C, priority=9, startTime=1680288900669; duration=0sec 2023-03-31 18:55:01,182 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:01,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:01,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288961236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:01,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 522 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288961236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:01,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288961538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:01,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 524 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288961538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,552 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/22090c38baff443a96b6a1e5d4150422 2023-03-31 18:55:01,970 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/0777c825d49b45e89482b5442dd8110d 2023-03-31 18:55:01,975 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/e4d3ace0400449599cfef3d0b618d5d8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e4d3ace0400449599cfef3d0b618d5d8 2023-03-31 18:55:01,978 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e4d3ace0400449599cfef3d0b618d5d8, entries=150, sequenceid=572, filesize=11.9 K 2023-03-31 18:55:01,979 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/22090c38baff443a96b6a1e5d4150422 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/22090c38baff443a96b6a1e5d4150422 2023-03-31 18:55:01,983 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/22090c38baff443a96b6a1e5d4150422, entries=150, sequenceid=572, filesize=11.9 K 2023-03-31 18:55:01,983 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/0777c825d49b45e89482b5442dd8110d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/0777c825d49b45e89482b5442dd8110d 2023-03-31 18:55:01,986 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/0777c825d49b45e89482b5442dd8110d, entries=150, sequenceid=572, filesize=11.9 K 2023-03-31 18:55:01,987 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2b9d344a6097fb5c7c41dc8319fac898 in 1260ms, sequenceid=572, compaction requested=false 2023-03-31 18:55:01,987 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:01,987 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:01,987 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:01,987 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:01,987 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,987 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:01,987 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:01,989 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:01,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:01,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174536a2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:01,991 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:01,991 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:01,992 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:01,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:01,992 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:01,992 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:01,992 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:01,992 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,992 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:01,994 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:01,994 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:01,994 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,994 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:01,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:01,996 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:01,996 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:01,996 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:01,996 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:01,996 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:01,998 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:01,998 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:01,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:01,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:01,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:01,999 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,001 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,007 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:02,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:02,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:02,013 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:02,013 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:02,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:02,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 18:55:02,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:02,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:02,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:02,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:02,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:02,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:02,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 534 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288962075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288962075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,113 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:02,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:02,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:02,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:02,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:02,116 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:02,116 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,116 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:02,116 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:02,117 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,117 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,118 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,118 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,119 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,119 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:02,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:02,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:02,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:02,120 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,120 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:02,120 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:02,120 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:02,122 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:02,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16853ff5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:02,123 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:02,124 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,130 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,130 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:02,130 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:02,130 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:02,130 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,130 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:02,131 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,132 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:02,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:02,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:02,134 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:02,134 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:02,134 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:02,134 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:02,134 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:02,135 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,135 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,135 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,136 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,137 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,139 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,139 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,139 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:02,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:02,139 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,139 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,139 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,139 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,139 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,139 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,139 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:02,139 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,139 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:02,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:02,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 536 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288962179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288962179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,239 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:02,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:02,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:02,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:02,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:02,243 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:02,243 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,243 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:02,243 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:02,243 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,243 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,245 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,245 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,246 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,246 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:02,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:02,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:02,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:02,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:02,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:02,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:02,247 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,247 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:02,247 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:02,247 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:02,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:02,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:02,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,249 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6346d648[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:02,250 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:02,251 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,252 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,252 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:02,252 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:02,252 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:02,252 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,252 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:02,253 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,253 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:02,253 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:02,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:02,256 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:02,256 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:02,256 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:02,256 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:02,256 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:02,261 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,261 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,261 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,262 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,265 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,265 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,265 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,265 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,265 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,265 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,265 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,265 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,265 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:02,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:02,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:02,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,266 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:02,266 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:02,366 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:02,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:02,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:02,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:02,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:02,369 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:02,369 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,369 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:02,369 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:02,370 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,370 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,371 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,371 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,371 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,371 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:02,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:02,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:02,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:02,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:02,373 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,373 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:02,373 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:02,373 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:02,376 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:02,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@235b0bd8[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:02,378 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:02,378 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,379 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,379 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:02,379 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,379 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:02,379 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:02,379 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:02,380 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:02,380 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,380 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:02,381 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 538 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288962382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:02,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:02,383 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:02,383 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:02,383 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:02,383 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:02,383 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:02,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288962383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,385 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,385 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,385 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,386 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,387 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,390 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:02,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:02,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,391 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:02,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,391 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:02,391 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:02,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/a569eb6ad40c4b7fa6888d5f036549b5 2023-03-31 18:55:02,491 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:02,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:02,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:02,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:02,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:02,496 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:02,496 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,496 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:02,496 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:02,497 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,497 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,498 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:02,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:02,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:02,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:02,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:02,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:02,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:02,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:02,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:02,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:02,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:02,502 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,502 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,502 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:02,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:02,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d15d937[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:02,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,505 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,505 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,505 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:02,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,506 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:02,506 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:02,506 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:02,508 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,508 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:02,508 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,508 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:02,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:02,510 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:02,510 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:02,510 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:02,510 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:02,510 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:02,512 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,512 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,512 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,516 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,518 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,521 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,521 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,521 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:02,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,521 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,521 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,521 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,521 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:02,522 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:02,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,521 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,522 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,522 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,522 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:02,622 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:02,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:02,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:02,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:02,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:02,625 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:02,625 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,625 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:02,625 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:02,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,627 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,627 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,628 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,628 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:02,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:02,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:02,629 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,629 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:02,629 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:02,629 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:02,632 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,632 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,632 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,632 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:02,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:02,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d027f7[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:02,634 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:02,634 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,636 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:02,636 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:02,636 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:02,636 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,636 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:02,637 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,637 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:02,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,637 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,638 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:02,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:02,641 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:02,641 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:02,641 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:02,641 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:02,641 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:02,642 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,642 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,643 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,644 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,650 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:02,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:02,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,651 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:02,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:02,651 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,651 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:02,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 540 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288962684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288962685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,751 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:02,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:02,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:02,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:02,756 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:02,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:02,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:02,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,758 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:02,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:02,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:02,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:02,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,759 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:02,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:02,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:02,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:02,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:02,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:02,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:02,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:02,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:02,762 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,762 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,762 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,762 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:02,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:02,763 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:02,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77714173[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:02,763 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,765 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,766 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,766 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:02,766 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:02,766 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:02,766 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:02,767 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,767 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:02,767 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,767 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:02,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:02,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:02,769 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:02,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:02,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:02,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:02,769 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:02,771 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,771 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,771 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:02,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:02,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,775 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:02,775 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:02,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/0ea29544a8ac401ea7d6d68c5739cbb7 2023-03-31 18:55:02,876 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:02,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:02,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:02,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:02,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:02,879 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:02,879 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,879 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:02,879 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:02,879 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,879 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,881 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,881 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,882 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,882 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:02,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:02,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:02,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:02,883 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:02,883 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:02,883 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:02,883 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:02,883 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:02,883 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,883 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:02,884 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:02,884 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:02,885 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,885 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,885 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,885 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:02,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c8f818e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:02,887 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:02,887 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,888 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,888 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:02,889 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,889 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:02,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:02,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,890 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:02,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:02,892 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:02,892 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:02,892 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:02,892 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:02,892 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:02,894 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,894 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:02,895 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,895 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,896 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:02,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,899 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,899 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:02,899 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,899 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:02,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:02,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:02,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:02,899 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:02,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:02,899 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,900 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:02,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,900 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:02,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:02,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:02,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:02,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288962951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,956 DEBUG [Thread-1525] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:02,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288962957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,960 DEBUG [Thread-1521] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:02,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:02,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288962960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:02,963 DEBUG [Thread-1523] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:03,000 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:03,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:03,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:03,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:03,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:03,003 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:03,003 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:03,003 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:03,003 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:03,004 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,004 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,005 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:03,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:03,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:03,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:03,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:03,006 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:03,006 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:03,006 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:03,006 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:03,006 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:03,006 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:03,007 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:03,007 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:03,007 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:03,007 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:03,007 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:03,007 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,007 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:03,007 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:03,009 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,009 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:03,009 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,009 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:03,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:03,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:03,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:03,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f602d49[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:03,011 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:03,011 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,013 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,014 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:03,014 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:03,014 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:03,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:03,015 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:03,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:03,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:03,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:03,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:03,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:03,019 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:03,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:03,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:03,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:03,019 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:03,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:03,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:03,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:03,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,025 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,025 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,025 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:03,025 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,026 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:03,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:03,026 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,026 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,026 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:03,025 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:03,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:03,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:03,026 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:03,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:03,026 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:03,126 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:03,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:03,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:03,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:03,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:03,130 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:03,130 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:03,130 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:03,130 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:03,130 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,130 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,141 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:03,141 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:03,142 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,142 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:03,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:03,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:03,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:03,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:03,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:03,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:03,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:03,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:03,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:03,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:03,144 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:03,144 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:03,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:03,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:03,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:03,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:03,147 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,147 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:03,147 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,147 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2c4acd[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:03,148 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:03,148 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,150 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:03,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:03,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:03,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:03,155 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:03,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:03,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:03,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:03,155 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:03,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:03,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,160 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,160 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,160 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:03,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:03,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:03,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:03,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:03,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:03,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:03,160 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:03,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,161 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:03,161 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:03,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:03,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288963190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:03,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 542 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288963192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,261 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:03,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:03,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:03,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:03,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:03,265 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:03,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:03,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:03,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:03,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,266 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,266 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:03,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:03,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:03,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:03,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:03,268 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:03,268 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:03,268 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:03,268 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:03,282 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/3ae604185fce4e75a53f96674bb865db 2023-03-31 18:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:03,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:03,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61abeae2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:03,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:03,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,286 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,286 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,286 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:03,288 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:03,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/a569eb6ad40c4b7fa6888d5f036549b5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a569eb6ad40c4b7fa6888d5f036549b5 2023-03-31 18:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:03,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:03,292 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:03,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:03,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:03,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:03,292 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:03,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a569eb6ad40c4b7fa6888d5f036549b5, entries=150, sequenceid=588, filesize=11.9 K 2023-03-31 18:55:03,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/0ea29544a8ac401ea7d6d68c5739cbb7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ea29544a8ac401ea7d6d68c5739cbb7 2023-03-31 18:55:03,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:03,295 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ea29544a8ac401ea7d6d68c5739cbb7, entries=150, sequenceid=588, filesize=11.9 K 2023-03-31 18:55:03,299 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/3ae604185fce4e75a53f96674bb865db as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3ae604185fce4e75a53f96674bb865db 2023-03-31 18:55:03,299 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:03,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:03,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:03,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:03,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:03,302 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:03,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,302 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:03,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:03,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:03,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:03,304 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3ae604185fce4e75a53f96674bb865db, entries=150, sequenceid=588, filesize=11.9 K 2023-03-31 18:55:03,305 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 2b9d344a6097fb5c7c41dc8319fac898 in 1259ms, sequenceid=588, compaction requested=true 2023-03-31 18:55:03,305 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:03,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:03,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:03,305 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:03,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:03,305 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:03,306 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:03,306 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:03,306 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/B is initiating minor compaction (all files) 2023-03-31 18:55:03,306 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/A is initiating minor compaction (all files) 2023-03-31 18:55:03,306 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/B in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:03,306 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/A in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:03,307 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/db1e884a99c64f458a4ebf18a6ad7854, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/22090c38baff443a96b6a1e5d4150422, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ea29544a8ac401ea7d6d68c5739cbb7] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.3 K 2023-03-31 18:55:03,307 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fc41c3582a7143f0911808d57c24aa6d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e4d3ace0400449599cfef3d0b618d5d8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a569eb6ad40c4b7fa6888d5f036549b5] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.3 K 2023-03-31 18:55:03,307 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting db1e884a99c64f458a4ebf18a6ad7854, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680288899877 2023-03-31 18:55:03,307 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting fc41c3582a7143f0911808d57c24aa6d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680288899877 2023-03-31 18:55:03,307 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 22090c38baff443a96b6a1e5d4150422, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680288900278 2023-03-31 18:55:03,307 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting e4d3ace0400449599cfef3d0b618d5d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680288900278 2023-03-31 18:55:03,307 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 0ea29544a8ac401ea7d6d68c5739cbb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680288900927 2023-03-31 18:55:03,308 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a569eb6ad40c4b7fa6888d5f036549b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680288900927 2023-03-31 18:55:03,325 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#B#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:03,329 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#A#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:03,349 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/10bdc80ca06a4220aabbfc8dd654b64e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10bdc80ca06a4220aabbfc8dd654b64e 2023-03-31 18:55:03,353 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/eca23551adbd4eeda57e6ef33c4ad9c9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/eca23551adbd4eeda57e6ef33c4ad9c9 2023-03-31 18:55:03,358 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/B of 2b9d344a6097fb5c7c41dc8319fac898 into 10bdc80ca06a4220aabbfc8dd654b64e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:03,358 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:03,358 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/B, priority=13, startTime=1680288903305; duration=0sec 2023-03-31 18:55:03,358 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:03,358 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:03,359 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/A of 2b9d344a6097fb5c7c41dc8319fac898 into eca23551adbd4eeda57e6ef33c4ad9c9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:03,359 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:03,359 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:03,359 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/A, priority=13, startTime=1680288903305; duration=0sec 2023-03-31 18:55:03,359 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/C is initiating minor compaction (all files) 2023-03-31 18:55:03,359 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:03,359 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/C in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:03,360 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/510a4dbe791c4b3c836172ce4334eb01, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/0777c825d49b45e89482b5442dd8110d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3ae604185fce4e75a53f96674bb865db] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.3 K 2023-03-31 18:55:03,360 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 510a4dbe791c4b3c836172ce4334eb01, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680288899877 2023-03-31 18:55:03,360 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 0777c825d49b45e89482b5442dd8110d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680288900278 2023-03-31 18:55:03,361 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3ae604185fce4e75a53f96674bb865db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680288900927 2023-03-31 18:55:03,367 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#C#compaction#245 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:03,376 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/4e4b5b9962d44d1d80c4fb2a1ebfead9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4e4b5b9962d44d1d80c4fb2a1ebfead9 2023-03-31 18:55:03,380 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/C of 2b9d344a6097fb5c7c41dc8319fac898 into 4e4b5b9962d44d1d80c4fb2a1ebfead9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:03,380 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:03,380 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/C, priority=13, startTime=1680288903305; duration=0sec 2023-03-31 18:55:03,380 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:03,403 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:03,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:03,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:03,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:03,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:03,407 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:03,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:03,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:03,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:03,408 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,408 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,409 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:03,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:03,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:03,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:03,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:03,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:03,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:03,411 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:03,411 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:03,411 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:55:03,411 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:03,412 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:03,412 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:03,412 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:03,412 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:03,412 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:03,819 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/d87e81b3bed44974a88c0c97bc66102b 2023-03-31 18:55:04,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:04,199 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:04,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:04,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 549 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288964211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:04,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288964212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,234 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/09d0cab94d6b46b3a54fdeb13a716ddb 2023-03-31 18:55:04,250 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/142425df736246c88c14d52a115292f3 2023-03-31 18:55:04,255 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/d87e81b3bed44974a88c0c97bc66102b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d87e81b3bed44974a88c0c97bc66102b 2023-03-31 18:55:04,259 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d87e81b3bed44974a88c0c97bc66102b, entries=150, sequenceid=611, filesize=11.9 K 2023-03-31 18:55:04,260 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/09d0cab94d6b46b3a54fdeb13a716ddb as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/09d0cab94d6b46b3a54fdeb13a716ddb 2023-03-31 18:55:04,264 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/09d0cab94d6b46b3a54fdeb13a716ddb, entries=150, sequenceid=611, filesize=11.9 K 2023-03-31 18:55:04,265 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/142425df736246c88c14d52a115292f3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/142425df736246c88c14d52a115292f3 2023-03-31 18:55:04,268 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/142425df736246c88c14d52a115292f3, entries=150, sequenceid=611, filesize=11.9 K 2023-03-31 18:55:04,269 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2b9d344a6097fb5c7c41dc8319fac898 in 858ms, sequenceid=611, compaction requested=false 2023-03-31 18:55:04,269 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:04,269 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:04,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:04,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:04,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:04,271 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:04,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d884a8[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:04,274 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:04,274 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,275 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,275 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:04,275 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:04,275 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:04,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:04,278 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:04,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:04,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:04,282 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:04,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:04,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:04,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:04,282 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:04,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,286 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:04,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:04,289 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:04,289 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:04,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:04,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:04,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 18:55:04,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:04,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:04,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:04,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:04,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:04,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:04,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:04,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 560 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288964349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:04,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288964349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,389 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:04,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:04,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:04,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:04,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:04,394 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:04,394 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:04,394 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:04,394 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:04,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,396 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,396 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,396 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,396 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:04,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:04,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:04,398 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,398 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:04,398 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:04,398 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:04,399 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:04,404 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:04,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f6ce9bf[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:04,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:04,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,407 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,407 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:04,407 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:04,407 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:04,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:04,408 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:04,408 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,409 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:04,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:04,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:04,412 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:04,412 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:04,412 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:04,412 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:04,412 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:04,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,417 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,419 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,421 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,422 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,422 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,422 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,422 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:04,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:04,422 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,422 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,422 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:04,422 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:04,422 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,422 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,422 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:04,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:04,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288964452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 562 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288964452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,522 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:04,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:04,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:04,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:04,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:04,525 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:04,525 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:04,526 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:04,526 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:04,526 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,526 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,527 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,527 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,528 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,528 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:04,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:04,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:04,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:04,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:04,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:04,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:04,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:04,531 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,531 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,531 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,531 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:04,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5168a4bd[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:04,533 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:04,533 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,535 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:04,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:04,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:04,535 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,535 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:04,536 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:04,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:04,539 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:04,539 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:04,539 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:04,539 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:04,539 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:04,540 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,540 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,540 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,541 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,542 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:04,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,544 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:04,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:04,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,544 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:04,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,545 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:04,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,645 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:04,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:04,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:04,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:04,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:04,648 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:04,648 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:04,648 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:04,648 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:04,649 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,649 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,651 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:04,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:04,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:04,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:04,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:04,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:04,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:04,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:04,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:04,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:04,654 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,654 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:04,654 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:04,654 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:04,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:04,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:04,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,657 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:04,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 564 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288964657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:04,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f2ab888[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:04,659 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:04,659 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:04,661 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288964659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,661 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:04,662 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:04,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:04,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:04,663 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:04,663 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,663 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,663 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:04,667 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:04,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:04,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:04,668 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:04,668 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:04,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,671 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:04,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:04,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,676 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:04,676 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:04,675 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,676 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,676 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,677 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,677 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,677 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/43797a8b25dd4deeb2394d244b5e7d29 2023-03-31 18:55:04,776 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:04,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:04,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:04,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:04,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:04,779 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:04,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:04,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:04,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:04,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,780 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,781 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,781 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,781 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:04,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:04,784 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,785 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,785 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:04,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a028c01[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:04,786 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:04,786 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,789 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:04,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:04,789 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:04,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:04,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,793 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,793 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,793 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:04,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:04,795 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:04,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:04,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:04,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:04,795 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:04,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,798 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,798 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,798 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,799 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,801 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,801 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,801 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,801 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,801 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:04,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:04,801 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,801 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,802 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:04,802 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:04,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:04,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,902 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:04,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:04,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:04,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:04,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:04,905 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:04,905 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:04,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:04,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:04,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,907 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,907 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,907 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:04,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:04,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:04,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:04,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:04,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:04,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:04,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:04,909 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,909 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:04,909 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:04,909 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:04,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:04,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:04,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:04,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:04,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ebabef[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:04,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,915 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:04,915 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:04,915 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:04,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:04,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,916 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:04,916 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,916 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:04,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:04,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:04,918 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:04,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:04,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:04,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:04,918 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:04,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:04,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:04,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:04,920 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,921 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:04,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:04,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:04,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:04,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:04,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:04,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:04,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:04,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:04,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:04,928 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:04,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:04,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:04,928 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:04,928 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:04,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:04,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 566 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288964960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:04,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:04,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288964963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,028 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:05,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:05,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:05,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:05,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:05,032 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:05,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:05,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:05,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,034 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,034 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:05,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:05,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:05,035 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,035 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:05,035 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:05,035 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:05,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:05,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:05,037 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:05,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6632a443[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:05,040 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:05,040 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,041 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:05,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:05,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:05,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:05,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:05,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:05,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:05,047 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:05,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:05,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:05,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:05,047 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:05,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:05,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:05,049 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:05,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,052 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,052 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,053 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,053 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,053 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:05,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,053 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:05,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:05,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:05,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:05,053 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,053 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,053 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,053 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:05,053 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:05,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:05,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/3b59000bb4f74c19a914c96d0d2233aa 2023-03-31 18:55:05,153 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:05,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:05,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:05,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:05,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:05,156 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:05,156 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:05,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:05,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,158 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:05,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:05,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:05,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:05,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:05,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:05,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:05,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:05,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ce6aef5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:05,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:05,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,166 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:05,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:05,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:05,166 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,166 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:05,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:05,167 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:05,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:05,170 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:05,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:05,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:05,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:05,170 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:05,174 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,176 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,178 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,178 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:05,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:05,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:05,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:05,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,179 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:05,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,179 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:05,179 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:05,279 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:05,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:05,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:05,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:05,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:05,283 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:05,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:05,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:05,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:05,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:05,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:05,286 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,286 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:05,286 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:05,286 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:05,288 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:05,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722a53ec[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:05,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:05,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,291 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:05,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:05,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:05,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:05,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:05,296 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:05,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:05,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:05,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:05,296 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:05,297 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,297 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:05,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,297 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:05,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,299 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:05,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:05,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:05,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,301 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:05,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:05,302 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:05,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:05,302 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:05,402 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:05,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:05,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:05,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:05,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:05,406 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:05,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:05,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:05,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,408 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,408 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,409 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,409 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:05,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:05,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:05,410 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,410 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:05,410 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:05,410 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:05,412 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:05,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:05,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db13a95[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:05,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,415 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:05,415 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:05,415 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:05,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:05,416 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:05,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:05,418 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:05,419 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:05,419 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:05,419 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:05,419 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:05,421 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,421 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,421 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:05,422 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,423 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,425 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,425 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,425 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:05,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:05,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,425 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:05,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:05,425 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:05,425 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:05,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:05,425 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,425 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:05,425 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,426 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,426 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,426 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:05,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 568 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288965464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:05,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52584 deadline: 1680288965465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,526 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:05,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:05,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:05,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:05,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:05,529 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:05,529 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,529 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:05,529 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:05,530 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,530 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,531 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,531 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,532 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,532 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:05,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:05,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:05,533 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,533 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:05,533 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:05,533 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,535 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:05,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ed29ddc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:05,537 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:05,537 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,540 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:05,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:05,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:05,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:05,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:05,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:05,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:05,545 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:05,545 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:05,545 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:05,545 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:05,545 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:05,546 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,546 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:05,546 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:05,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:05,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,549 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:05,551 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/fa5e3bb9c4cd42d0bb8df342ad9c5f3e 2023-03-31 18:55:05,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/43797a8b25dd4deeb2394d244b5e7d29 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/43797a8b25dd4deeb2394d244b5e7d29 2023-03-31 18:55:05,559 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,559 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:05,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:05,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:05,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:05,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:05,559 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,560 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:05,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:05,560 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,560 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:05,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:05,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/43797a8b25dd4deeb2394d244b5e7d29, entries=250, sequenceid=625, filesize=16.7 K 2023-03-31 18:55:05,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/3b59000bb4f74c19a914c96d0d2233aa as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3b59000bb4f74c19a914c96d0d2233aa 2023-03-31 18:55:05,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3b59000bb4f74c19a914c96d0d2233aa, entries=150, sequenceid=625, filesize=11.9 K 2023-03-31 18:55:05,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/fa5e3bb9c4cd42d0bb8df342ad9c5f3e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/fa5e3bb9c4cd42d0bb8df342ad9c5f3e 2023-03-31 18:55:05,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/fa5e3bb9c4cd42d0bb8df342ad9c5f3e, entries=150, sequenceid=625, filesize=11.9 K 2023-03-31 18:55:05,571 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 2b9d344a6097fb5c7c41dc8319fac898 in 1255ms, sequenceid=625, compaction requested=true 2023-03-31 18:55:05,571 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:05,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:05,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:05,572 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:05,572 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:05,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:05,573 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42188 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:05,573 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:05,573 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/A is initiating minor compaction (all files) 2023-03-31 18:55:05,573 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/B is initiating minor compaction (all files) 2023-03-31 18:55:05,573 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/A in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,573 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/B in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,573 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/eca23551adbd4eeda57e6ef33c4ad9c9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d87e81b3bed44974a88c0c97bc66102b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/43797a8b25dd4deeb2394d244b5e7d29] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=41.2 K 2023-03-31 18:55:05,573 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10bdc80ca06a4220aabbfc8dd654b64e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/09d0cab94d6b46b3a54fdeb13a716ddb, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3b59000bb4f74c19a914c96d0d2233aa] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.4 K 2023-03-31 18:55:05,573 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting eca23551adbd4eeda57e6ef33c4ad9c9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680288900927 2023-03-31 18:55:05,574 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 10bdc80ca06a4220aabbfc8dd654b64e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680288900927 2023-03-31 18:55:05,574 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d87e81b3bed44974a88c0c97bc66102b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1680288902064 2023-03-31 18:55:05,574 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 09d0cab94d6b46b3a54fdeb13a716ddb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1680288902064 2023-03-31 18:55:05,574 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 43797a8b25dd4deeb2394d244b5e7d29, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680288904201 2023-03-31 18:55:05,575 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3b59000bb4f74c19a914c96d0d2233aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680288904211 2023-03-31 18:55:05,585 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#A#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:05,601 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#B#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:05,627 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/bd299f8955644d55bd233a4b565c4895 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/bd299f8955644d55bd233a4b565c4895 2023-03-31 18:55:05,635 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/B of 2b9d344a6097fb5c7c41dc8319fac898 into bd299f8955644d55bd233a4b565c4895(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:05,635 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:05,635 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/B, priority=13, startTime=1680288905572; duration=0sec 2023-03-31 18:55:05,635 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:05,635 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:05,636 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:05,637 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/C is initiating minor compaction (all files) 2023-03-31 18:55:05,637 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/C in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,637 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4e4b5b9962d44d1d80c4fb2a1ebfead9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/142425df736246c88c14d52a115292f3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/fa5e3bb9c4cd42d0bb8df342ad9c5f3e] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.4 K 2023-03-31 18:55:05,637 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4e4b5b9962d44d1d80c4fb2a1ebfead9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680288900927 2023-03-31 18:55:05,638 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 142425df736246c88c14d52a115292f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1680288902064 2023-03-31 18:55:05,638 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting fa5e3bb9c4cd42d0bb8df342ad9c5f3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680288904211 2023-03-31 18:55:05,645 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#C#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:05,660 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:05,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:05,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:05,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:05,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:05,663 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:05,663 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,663 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:05,663 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:05,663 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,663 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:05,664 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,665 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:05,665 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:05,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:05,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:05,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:05,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:05,666 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:05,666 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:05,666 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:05,667 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:05,667 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:05,667 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:05,667 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:05,667 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-31 18:55:05,667 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:05,667 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:05,667 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:05,667 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:05,667 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:05,667 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:06,026 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/d2bbff5e64d44c659a61bb8cb8848ff4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d2bbff5e64d44c659a61bb8cb8848ff4 2023-03-31 18:55:06,030 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/A of 2b9d344a6097fb5c7c41dc8319fac898 into d2bbff5e64d44c659a61bb8cb8848ff4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:06,030 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:06,030 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/A, priority=13, startTime=1680288905572; duration=0sec 2023-03-31 18:55:06,030 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:06,060 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/39f25991b2ed4919a34b774a85edd4f2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/39f25991b2ed4919a34b774a85edd4f2 2023-03-31 18:55:06,065 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/C of 2b9d344a6097fb5c7c41dc8319fac898 into 39f25991b2ed4919a34b774a85edd4f2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:06,065 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:06,065 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/C, priority=13, startTime=1680288905572; duration=0sec 2023-03-31 18:55:06,065 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:06,082 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/2288b59afe0c4a8ca8e7755713b4206a 2023-03-31 18:55:06,093 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/aadb64b8c8bc4a848cfc69e9751d2be7 2023-03-31 18:55:06,113 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/5e0765517f574d339423420665be730a 2023-03-31 18:55:06,120 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/2288b59afe0c4a8ca8e7755713b4206a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/2288b59afe0c4a8ca8e7755713b4206a 2023-03-31 18:55:06,134 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/2288b59afe0c4a8ca8e7755713b4206a, entries=150, sequenceid=645, filesize=11.9 K 2023-03-31 18:55:06,135 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/aadb64b8c8bc4a848cfc69e9751d2be7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/aadb64b8c8bc4a848cfc69e9751d2be7 2023-03-31 18:55:06,139 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/aadb64b8c8bc4a848cfc69e9751d2be7, entries=150, sequenceid=645, filesize=11.9 K 2023-03-31 18:55:06,139 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/5e0765517f574d339423420665be730a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/5e0765517f574d339423420665be730a 2023-03-31 18:55:06,143 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/5e0765517f574d339423420665be730a, entries=150, sequenceid=645, filesize=11.9 K 2023-03-31 18:55:06,144 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 2b9d344a6097fb5c7c41dc8319fac898 in 477ms, sequenceid=645, compaction requested=false 2023-03-31 18:55:06,144 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:06,144 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:06,146 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,146 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,146 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,146 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:06,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47aa0ff5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:06,148 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:06,148 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,149 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,149 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:06,149 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:06,149 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:06,149 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,149 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:06,152 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:06,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:06,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:06,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:06,155 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:06,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:06,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:06,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:06,155 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:06,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:06,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:06,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:06,162 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:06,162 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:06,262 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:06,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:06,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:06,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:06,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:06,266 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:06,266 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,266 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:06,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:06,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,268 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,269 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,269 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,269 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:06,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:06,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:06,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:06,270 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:06,270 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,270 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:06,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:06,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:06,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:06,273 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,274 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,274 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:06,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@673f91df[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:06,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:06,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,276 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:06,277 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,277 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:06,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:06,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,278 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:06,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:06,281 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:06,281 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:06,281 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:06,281 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:06,281 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:06,283 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,283 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,283 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,288 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,288 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:06,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:06,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,289 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:06,289 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:06,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:06,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,389 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:06,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:06,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:06,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:06,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:06,393 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:06,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:06,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:06,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,394 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,394 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:06,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:06,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:06,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:06,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:06,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:06,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:06,398 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:06,400 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:06,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@338e7a3f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:06,400 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,401 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,401 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:06,401 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:06,401 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:06,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:06,404 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:06,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:06,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:06,407 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:06,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:06,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:06,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:06,407 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:06,408 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,408 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,408 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,409 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:06,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:06,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:06,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,415 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:06,416 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:06,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,416 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:06,475 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:55:06,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:06,476 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:06,476 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:06,476 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:06,476 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:06,476 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:06,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/9065287e06a04ff7b7c593cb9b4f2931 2023-03-31 18:55:06,491 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/684b22d3edd44ac3bd22a5a68f3af969 2023-03-31 18:55:06,500 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/31444617951f42caa672ca134fa4214c 2023-03-31 18:55:06,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/9065287e06a04ff7b7c593cb9b4f2931 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9065287e06a04ff7b7c593cb9b4f2931 2023-03-31 18:55:06,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288906492, max timestamp=1680288906505Num uniques 859; 2023-03-31 18:55:06,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:55:06,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288906492, max timestamp=1680288906505Num uniques 859; 2023-03-31 18:55:06,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:55:06,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288906492, max timestamp=1680288906505Num uniques 859; 2023-03-31 18:55:06,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:55:06,508 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9065287e06a04ff7b7c593cb9b4f2931, entries=200, sequenceid=659, filesize=14.3 K 2023-03-31 18:55:06,516 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:06,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:06,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:06,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:06,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:06,519 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:06,519 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,519 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:06,519 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:06,520 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,520 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/684b22d3edd44ac3bd22a5a68f3af969 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/684b22d3edd44ac3bd22a5a68f3af969 2023-03-31 18:55:06,522 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,522 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,522 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,522 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:06,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:06,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:06,524 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:06,524 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,524 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:06,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/684b22d3edd44ac3bd22a5a68f3af969, entries=150, sequenceid=659, filesize=11.9 K 2023-03-31 18:55:06,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/31444617951f42caa672ca134fa4214c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/31444617951f42caa672ca134fa4214c 2023-03-31 18:55:06,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:06,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:06,526 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,526 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:06,526 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:06,526 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:06,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:06,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:06,527 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:06,528 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:06,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/31444617951f42caa672ca134fa4214c, entries=150, sequenceid=659, filesize=11.9 K 2023-03-31 18:55:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=107.34 KB/109920 for 2b9d344a6097fb5c7c41dc8319fac898 in 54ms, sequenceid=659, compaction requested=true 2023-03-31 18:55:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:06,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:06,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:06,530 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:06,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,530 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:06,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9083def[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:06,530 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:06,530 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,530 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:06,531 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/B is initiating minor compaction (all files) 2023-03-31 18:55:06,531 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/B in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,531 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/bd299f8955644d55bd233a4b565c4895, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/aadb64b8c8bc4a848cfc69e9751d2be7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/684b22d3edd44ac3bd22a5a68f3af969] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.5 K 2023-03-31 18:55:06,531 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39850 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:06,531 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/A is initiating minor compaction (all files) 2023-03-31 18:55:06,531 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/A in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,531 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d2bbff5e64d44c659a61bb8cb8848ff4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/2288b59afe0c4a8ca8e7755713b4206a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9065287e06a04ff7b7c593cb9b4f2931] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=38.9 K 2023-03-31 18:55:06,531 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting bd299f8955644d55bd233a4b565c4895, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680288904211 2023-03-31 18:55:06,532 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,532 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:06,532 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:06,532 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:06,533 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d2bbff5e64d44c659a61bb8cb8848ff4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680288904211 2023-03-31 18:55:06,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:06,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=114.05 KB heapSize=325.22 KB 2023-03-31 18:55:06,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:06,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:06,534 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 2288b59afe0c4a8ca8e7755713b4206a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1680288904343 2023-03-31 18:55:06,535 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting aadb64b8c8bc4a848cfc69e9751d2be7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1680288904343 2023-03-31 18:55:06,535 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,535 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:06,535 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:06,535 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:06,535 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:06,535 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:06,535 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9065287e06a04ff7b7c593cb9b4f2931, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1680288906468 2023-03-31 18:55:06,535 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:06,535 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:06,536 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 684b22d3edd44ac3bd22a5a68f3af969, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1680288906472 2023-03-31 18:55:06,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:06,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:06,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:06,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:06,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:06,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:06,540 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,540 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,540 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,541 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,543 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,546 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#A#compaction#262 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:06,547 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#B#compaction#263 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:06,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:06,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:06,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,548 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:06,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:06,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,549 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:06,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,549 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:06,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52584 deadline: 1680288966564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:06,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 597 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288966564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,577 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/e6fa9312be564fd9a401fcb922514a36 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e6fa9312be564fd9a401fcb922514a36 2023-03-31 18:55:06,577 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/71210211238b4a48815d0e67c656b0f2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/71210211238b4a48815d0e67c656b0f2 2023-03-31 18:55:06,582 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/A of 2b9d344a6097fb5c7c41dc8319fac898 into e6fa9312be564fd9a401fcb922514a36(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:06,582 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:06,582 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/A, priority=13, startTime=1680288906529; duration=0sec 2023-03-31 18:55:06,582 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:06,582 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:06,583 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/B of 2b9d344a6097fb5c7c41dc8319fac898 into 71210211238b4a48815d0e67c656b0f2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:06,583 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:06,583 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/B, priority=13, startTime=1680288906530; duration=0sec 2023-03-31 18:55:06,583 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:06,583 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:06,583 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/C is initiating minor compaction (all files) 2023-03-31 18:55:06,583 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/C in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,584 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/39f25991b2ed4919a34b774a85edd4f2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/5e0765517f574d339423420665be730a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/31444617951f42caa672ca134fa4214c] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.5 K 2023-03-31 18:55:06,584 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 39f25991b2ed4919a34b774a85edd4f2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680288904211 2023-03-31 18:55:06,585 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 5e0765517f574d339423420665be730a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1680288904343 2023-03-31 18:55:06,585 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 31444617951f42caa672ca134fa4214c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1680288906472 2023-03-31 18:55:06,593 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#C#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:06,649 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:06,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:06,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:06,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:06,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:06,652 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:06,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:06,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:06,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,654 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,654 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,654 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,654 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:06,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:06,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:06,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:06,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:06,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:06,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:06,656 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,656 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:06,656 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:06,656 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:06,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:06,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:06,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:06,657 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,658 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,658 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,658 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:06,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4a5e9c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:06,659 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:06,659 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,661 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,661 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:06,661 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,661 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:06,661 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:06,661 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:06,662 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:06,663 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,663 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:06,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:06,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:06,667 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:06,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:06,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:06,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:06,667 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:06,669 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,669 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,669 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,670 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:06,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52584 deadline: 1680288966671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:06,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 599 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288966671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:06,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:06,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,674 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:06,674 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:06,673 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,674 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,674 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,774 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:06,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:06,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:06,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:06,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:06,777 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:06,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:06,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:06,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,780 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,780 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,780 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:06,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:06,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:06,781 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,781 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:06,781 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:06,781 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:06,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:06,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e085e3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:06,785 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:06,785 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,786 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:06,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:06,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:06,786 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,786 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:06,787 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:06,787 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,787 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:06,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:06,789 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:06,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:06,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:06,790 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:06,790 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:06,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,792 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,792 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,793 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:06,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:06,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:06,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,795 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:06,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,795 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:06,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:06,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52584 deadline: 1680288966874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:06,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 601 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288966874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,896 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:06,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:06,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:06,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:06,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:06,899 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:06,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:06,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:06,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:06,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:06,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:06,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:06,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:06,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:06,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:06,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:06,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:06,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:06,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:06,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:06,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:06,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:06,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:06,907 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,908 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,908 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,908 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:06,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ff58e2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:06,910 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:06,910 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,911 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,911 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:06,911 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:06,911 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:06,911 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,911 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:06,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:06,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,913 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:06,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:06,915 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:06,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:06,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:06,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:06,915 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:06,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:06,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:06,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:06,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:06,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:06,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:06,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:06,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:06,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:06,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:06,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:06,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:06,922 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:06,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:06,922 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:06,922 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:06,948 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/a73e161c140141dcb7d4987637d1c14d 2023-03-31 18:55:06,959 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/46162eba2ece4ed9b70ba8a57e6bf107 2023-03-31 18:55:06,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:06,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52502 deadline: 1680288966961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,965 DEBUG [Thread-1525] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:06,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:06,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52508 deadline: 1680288966966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:06,968 DEBUG [Thread-1521] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:06,970 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/d746d701dd734747927a713945218b69 2023-03-31 18:55:06,974 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/a73e161c140141dcb7d4987637d1c14d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a73e161c140141dcb7d4987637d1c14d 2023-03-31 18:55:06,977 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a73e161c140141dcb7d4987637d1c14d, entries=150, sequenceid=694, filesize=11.9 K 2023-03-31 18:55:06,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/46162eba2ece4ed9b70ba8a57e6bf107 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/46162eba2ece4ed9b70ba8a57e6bf107 2023-03-31 18:55:06,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/46162eba2ece4ed9b70ba8a57e6bf107, entries=150, sequenceid=694, filesize=11.9 K 2023-03-31 18:55:06,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/d746d701dd734747927a713945218b69 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d746d701dd734747927a713945218b69 2023-03-31 18:55:06,985 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d746d701dd734747927a713945218b69, entries=150, sequenceid=694, filesize=11.9 K 2023-03-31 18:55:06,986 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~345.09 KB/353376, currentSize=60.38 KB/61830 for 2b9d344a6097fb5c7c41dc8319fac898 in 452ms, sequenceid=694, compaction requested=false 2023-03-31 18:55:06,986 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:06,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:06,995 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 18:55:06,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:06,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:06,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:06,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:06,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:06,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:07,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/9410292f8b794906bee25553cd8913fa 2023-03-31 18:55:07,023 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:07,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:07,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:07,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:07,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:07,028 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:07,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:07,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:07,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:07,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,030 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,030 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:07,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:07,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:07,030 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,030 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:07,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:07,035 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/714d6b95f39842ac910930861a34ac53 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/714d6b95f39842ac910930861a34ac53 2023-03-31 18:55:07,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:07,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:07,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:07,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:07,036 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:07,040 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:07,040 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:07,040 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/C of 2b9d344a6097fb5c7c41dc8319fac898 into 714d6b95f39842ac910930861a34ac53(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:07,040 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:07,040 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/C, priority=13, startTime=1680288906530; duration=0sec 2023-03-31 18:55:07,040 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:07,040 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,040 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:07,041 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:07,041 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:07,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:07,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd0f2f7[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:07,046 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:07,046 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,047 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:07,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:07,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:07,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:07,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:07,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:07,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:07,053 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:07,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:07,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:07,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:07,053 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:07,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:07,056 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,057 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,059 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,059 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:07,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:07,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:07,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:07,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:07,059 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:07,059 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:07,060 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:07,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:07,059 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:07,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:07,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:07,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288967062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,160 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:07,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:07,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:07,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:07,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:07,163 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:07,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:07,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:07,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:07,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,165 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:07,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:07,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:07,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:07,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:07,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:07,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:07,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,166 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288967166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:07,167 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:07,167 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:07,167 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:07,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:07,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:07,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:07,170 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:07,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:07,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4122e36f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:07,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:07,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,173 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:07,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:07,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:07,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:07,174 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:07,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:07,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:07,177 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:07,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:07,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:07,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:07,177 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:07,178 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,178 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,178 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:07,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52584 deadline: 1680288967177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 603 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288967178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:07,179 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:07,188 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:07,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:07,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:07,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:07,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:07,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:07,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,189 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,189 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,189 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:07,189 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:07,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:07,289 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:07,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:07,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:07,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:07,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:07,292 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:07,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:07,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:07,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:07,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,293 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:07,293 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:07,294 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,294 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:07,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:07,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:07,295 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,295 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:07,295 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:07,295 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:07,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:07,297 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,297 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,297 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,297 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40eca935[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:07,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:07,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,302 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:07,302 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:07,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:07,302 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:07,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:07,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:07,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:07,306 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:07,306 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:07,306 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:07,306 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:07,306 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:07,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:07,309 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:07,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:07,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:07,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:07,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:07,312 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:07,312 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:07,312 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:07,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:07,312 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:07,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:07,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:07,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288967368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,412 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:07,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:07,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:07,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:07,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:07,415 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:07,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:07,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:07,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:07,416 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,416 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,418 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:07,418 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:07,418 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,418 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:07,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:07,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:07,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:07,420 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,420 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:07,420 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:07,420 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:07,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:07,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:07,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:07,421 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,421 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,421 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,421 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:07,423 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:07,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f6bc182[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:07,423 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,424 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:07,424 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:07,424 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:07,424 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:07,425 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:07,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:07,427 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:07,427 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:07,427 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:07,427 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:07,427 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:07,429 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,429 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:07,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,430 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:07,430 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,431 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,433 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,433 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:07,433 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,433 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:07,433 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:07,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:07,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:07,433 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:07,433 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,434 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:07,434 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:07,434 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:07,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:07,434 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/3211db8e9140454a911524a8f375c878 2023-03-31 18:55:07,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/3baa179f60e94885ab0b9425891f901f 2023-03-31 18:55:07,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/9410292f8b794906bee25553cd8913fa as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9410292f8b794906bee25553cd8913fa 2023-03-31 18:55:07,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9410292f8b794906bee25553cd8913fa, entries=200, sequenceid=709, filesize=14.3 K 2023-03-31 18:55:07,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/3211db8e9140454a911524a8f375c878 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3211db8e9140454a911524a8f375c878 2023-03-31 18:55:07,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3211db8e9140454a911524a8f375c878, entries=150, sequenceid=709, filesize=11.9 K 2023-03-31 18:55:07,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/3baa179f60e94885ab0b9425891f901f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3baa179f60e94885ab0b9425891f901f 2023-03-31 18:55:07,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3baa179f60e94885ab0b9425891f901f, entries=150, sequenceid=709, filesize=11.9 K 2023-03-31 18:55:07,470 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2b9d344a6097fb5c7c41dc8319fac898 in 475ms, sequenceid=709, compaction requested=true 2023-03-31 18:55:07,470 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:07,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:07,470 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:07,470 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:07,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:07,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:07,471 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:07,471 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/A is initiating minor compaction (all files) 2023-03-31 18:55:07,471 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:07,471 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/A in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,471 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/B is initiating minor compaction (all files) 2023-03-31 18:55:07,471 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e6fa9312be564fd9a401fcb922514a36, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a73e161c140141dcb7d4987637d1c14d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9410292f8b794906bee25553cd8913fa] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=39.0 K 2023-03-31 18:55:07,471 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/B in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,471 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/71210211238b4a48815d0e67c656b0f2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/46162eba2ece4ed9b70ba8a57e6bf107, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3211db8e9140454a911524a8f375c878] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.6 K 2023-03-31 18:55:07,472 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting e6fa9312be564fd9a401fcb922514a36, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1680288906472 2023-03-31 18:55:07,472 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 71210211238b4a48815d0e67c656b0f2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1680288906472 2023-03-31 18:55:07,472 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a73e161c140141dcb7d4987637d1c14d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1680288906527 2023-03-31 18:55:07,472 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 46162eba2ece4ed9b70ba8a57e6bf107, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1680288906527 2023-03-31 18:55:07,472 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9410292f8b794906bee25553cd8913fa, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1680288906550 2023-03-31 18:55:07,473 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3211db8e9140454a911524a8f375c878, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1680288906550 2023-03-31 18:55:07,486 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#B#compaction#270 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:07,493 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#A#compaction#271 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:07,534 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:07,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:07,535 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/70e228ca05a94e2eb7834be544d18871 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/70e228ca05a94e2eb7834be544d18871 2023-03-31 18:55:07,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:07,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:07,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:07,538 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:07,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:07,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:07,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:07,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,539 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:07,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:07,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:07,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:07,541 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/A of 2b9d344a6097fb5c7c41dc8319fac898 into 70e228ca05a94e2eb7834be544d18871(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:07,542 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:07,542 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/A, priority=13, startTime=1680288907470; duration=0sec 2023-03-31 18:55:07,542 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:07,542 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:07,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:07,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:07,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:07,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:07,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:07,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:07,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:07,542 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,543 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. started... 2023-03-31 18:55:07,543 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-31 18:55:07,543 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:07,543 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/C is initiating minor compaction (all files) 2023-03-31 18:55:07,543 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:07,543 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/C in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:07,543 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:07,543 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/714d6b95f39842ac910930861a34ac53, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d746d701dd734747927a713945218b69, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3baa179f60e94885ab0b9425891f901f] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.6 K 2023-03-31 18:55:07,543 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:07,543 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:07,543 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:07,544 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:07,544 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 714d6b95f39842ac910930861a34ac53, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1680288906472 2023-03-31 18:55:07,544 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d746d701dd734747927a713945218b69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1680288906527 2023-03-31 18:55:07,544 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 3baa179f60e94885ab0b9425891f901f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1680288906550 2023-03-31 18:55:07,553 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#C#compaction#273 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:07,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:07,674 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. as already flushing 2023-03-31 18:55:07,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288967692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 607 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288967692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52584 deadline: 1680288967694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288967797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 609 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288967800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:07,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52584 deadline: 1680288967801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:07,933 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/10cbf25c61dd46ffbf674f1d106ce8a6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10cbf25c61dd46ffbf674f1d106ce8a6 2023-03-31 18:55:07,938 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/B of 2b9d344a6097fb5c7c41dc8319fac898 into 10cbf25c61dd46ffbf674f1d106ce8a6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:07,938 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:07,939 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/B, priority=13, startTime=1680288907470; duration=0sec 2023-03-31 18:55:07,939 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:07,979 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/1bf694edeec14a72afd3e48c73f6dd58 2023-03-31 18:55:07,989 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/1b41777dc8204b4094bb91edfb29454a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/1b41777dc8204b4094bb91edfb29454a 2023-03-31 18:55:07,999 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/C of 2b9d344a6097fb5c7c41dc8319fac898 into 1b41777dc8204b4094bb91edfb29454a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:08,000 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:08,000 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/C, priority=13, startTime=1680288907470; duration=0sec 2023-03-31 18:55:08,000 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:08,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:08,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52518 deadline: 1680288968001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:08,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 611 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52530 deadline: 1680288968003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:08,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52584 deadline: 1680288968006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,024 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/feb4d5ea0434423fb5d30bcb99cc96b1 2023-03-31 18:55:08,050 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/a0bd5cd77f1e41928b34150274373a26 2023-03-31 18:55:08,058 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/1bf694edeec14a72afd3e48c73f6dd58 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1bf694edeec14a72afd3e48c73f6dd58 2023-03-31 18:55:08,068 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1bf694edeec14a72afd3e48c73f6dd58, entries=150, sequenceid=730, filesize=11.9 K 2023-03-31 18:55:08,069 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/feb4d5ea0434423fb5d30bcb99cc96b1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/feb4d5ea0434423fb5d30bcb99cc96b1 2023-03-31 18:55:08,086 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/feb4d5ea0434423fb5d30bcb99cc96b1, entries=150, sequenceid=730, filesize=11.9 K 2023-03-31 18:55:08,087 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/a0bd5cd77f1e41928b34150274373a26 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/a0bd5cd77f1e41928b34150274373a26 2023-03-31 18:55:08,093 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/a0bd5cd77f1e41928b34150274373a26, entries=150, sequenceid=730, filesize=11.9 K 2023-03-31 18:55:08,097 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2b9d344a6097fb5c7c41dc8319fac898 in 553ms, sequenceid=730, compaction requested=false 2023-03-31 18:55:08,097 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:08,097 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:08,099 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:08,099 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:08,100 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:08,100 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:08,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:08,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:08,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:08,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:08,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7526c3d2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:08,102 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:08,102 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:08,103 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:08,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:08,104 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:08,104 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:08,104 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:08,104 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,104 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:08,105 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:08,105 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:08,105 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:08,105 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:08,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:08,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:08,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:08,110 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:08,110 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:08,110 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:08,110 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:08,110 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:08,112 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:08,112 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:08,112 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:08,113 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:08,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:08,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,117 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,120 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,120 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:08,120 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:08,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:08,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:08,120 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:08,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:08,121 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:08,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:08,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:08,120 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:08,121 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:08,121 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:08,121 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:08,121 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:08,121 DEBUG [Thread-1531] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:08,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:08,121 DEBUG [Thread-1531] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:08,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:08,190 DEBUG [Thread-1532] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:63858 2023-03-31 18:55:08,190 DEBUG [Thread-1532] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:08,191 DEBUG [Thread-1536] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:63858 2023-03-31 18:55:08,191 DEBUG [Thread-1536] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:08,191 DEBUG [Thread-1538] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:63858 2023-03-31 18:55:08,191 DEBUG [Thread-1538] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:08,192 DEBUG [Thread-1534] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:63858 2023-03-31 18:55:08,192 DEBUG [Thread-1534] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:08,192 DEBUG [Thread-1540] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de064cb to 127.0.0.1:63858 2023-03-31 18:55:08,192 DEBUG [Thread-1540] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:08,221 DEBUG [Thread-1531] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:08,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:08,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:08,313 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-31 18:55:08,313 DEBUG [Thread-1529] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7810a231 to 127.0.0.1:63858 2023-03-31 18:55:08,313 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:08,313 DEBUG [Thread-1529] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:08,313 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:08,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:08,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:08,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:08,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:08,314 DEBUG [Thread-1523] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:63858 2023-03-31 18:55:08,314 DEBUG [Thread-1523] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:08,314 DEBUG [Thread-1527] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:63858 2023-03-31 18:55:08,314 DEBUG [Thread-1527] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:08,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/52ae0f8bf11844dc8170b97cb3bc3f21 2023-03-31 18:55:08,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/b0e5ae2e8a7f4a04a442a72e29e13836 2023-03-31 18:55:08,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/60ac70aeae6644bead87b7ede97e303b 2023-03-31 18:55:08,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/52ae0f8bf11844dc8170b97cb3bc3f21 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/52ae0f8bf11844dc8170b97cb3bc3f21 2023-03-31 18:55:08,744 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/52ae0f8bf11844dc8170b97cb3bc3f21, entries=150, sequenceid=748, filesize=11.9 K 2023-03-31 18:55:08,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/b0e5ae2e8a7f4a04a442a72e29e13836 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b0e5ae2e8a7f4a04a442a72e29e13836 2023-03-31 18:55:08,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b0e5ae2e8a7f4a04a442a72e29e13836, entries=150, sequenceid=748, filesize=11.9 K 2023-03-31 18:55:08,748 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/60ac70aeae6644bead87b7ede97e303b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/60ac70aeae6644bead87b7ede97e303b 2023-03-31 18:55:08,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/60ac70aeae6644bead87b7ede97e303b, entries=150, sequenceid=748, filesize=11.9 K 2023-03-31 18:55:08,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 2b9d344a6097fb5c7c41dc8319fac898 in 439ms, sequenceid=748, compaction requested=true 2023-03-31 18:55:08,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:08,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:08,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:08,752 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:08,752 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:08,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:08,753 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:08,753 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:08,753 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/A is initiating minor compaction (all files) 2023-03-31 18:55:08,753 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/B is initiating minor compaction (all files) 2023-03-31 18:55:08,753 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/A in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:08,753 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/B in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:08,753 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/70e228ca05a94e2eb7834be544d18871, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1bf694edeec14a72afd3e48c73f6dd58, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/52ae0f8bf11844dc8170b97cb3bc3f21] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.7 K 2023-03-31 18:55:08,753 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10cbf25c61dd46ffbf674f1d106ce8a6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/feb4d5ea0434423fb5d30bcb99cc96b1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b0e5ae2e8a7f4a04a442a72e29e13836] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.7 K 2023-03-31 18:55:08,754 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 70e228ca05a94e2eb7834be544d18871, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1680288906550 2023-03-31 18:55:08,754 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 10cbf25c61dd46ffbf674f1d106ce8a6, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1680288906550 2023-03-31 18:55:08,754 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 1bf694edeec14a72afd3e48c73f6dd58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=730, earliestPutTs=1680288907052 2023-03-31 18:55:08,754 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting feb4d5ea0434423fb5d30bcb99cc96b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=730, earliestPutTs=1680288907052 2023-03-31 18:55:08,754 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 52ae0f8bf11844dc8170b97cb3bc3f21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=748, earliestPutTs=1680288907691 2023-03-31 18:55:08,754 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b0e5ae2e8a7f4a04a442a72e29e13836, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=748, earliestPutTs=1680288907691 2023-03-31 18:55:08,762 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#A#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:08,762 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#B#compaction#280 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:08,771 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/409939c0383a43969920ec876a771df3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/409939c0383a43969920ec876a771df3 2023-03-31 18:55:08,775 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/B of 2b9d344a6097fb5c7c41dc8319fac898 into 409939c0383a43969920ec876a771df3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:08,775 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:08,775 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/B, priority=13, startTime=1680288908752; duration=0sec 2023-03-31 18:55:08,775 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:08,775 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:08,776 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:08,776 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 2b9d344a6097fb5c7c41dc8319fac898/C is initiating minor compaction (all files) 2023-03-31 18:55:08,776 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b9d344a6097fb5c7c41dc8319fac898/C in TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:08,776 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/1b41777dc8204b4094bb91edfb29454a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/a0bd5cd77f1e41928b34150274373a26, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/60ac70aeae6644bead87b7ede97e303b] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp, totalSize=36.7 K 2023-03-31 18:55:08,776 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 1b41777dc8204b4094bb91edfb29454a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1680288906550 2023-03-31 18:55:08,776 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a0bd5cd77f1e41928b34150274373a26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=730, earliestPutTs=1680288907052 2023-03-31 18:55:08,777 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 60ac70aeae6644bead87b7ede97e303b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=748, earliestPutTs=1680288907691 2023-03-31 18:55:08,781 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b9d344a6097fb5c7c41dc8319fac898#C#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:09,173 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/5fe2a4c02a684f119ab5663e44a404bf as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/5fe2a4c02a684f119ab5663e44a404bf 2023-03-31 18:55:09,176 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/A of 2b9d344a6097fb5c7c41dc8319fac898 into 5fe2a4c02a684f119ab5663e44a404bf(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:09,176 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:09,176 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/A, priority=13, startTime=1680288908752; duration=0sec 2023-03-31 18:55:09,176 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:09,189 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/d609230f372b4d9b86575d6f75a40873 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d609230f372b4d9b86575d6f75a40873 2023-03-31 18:55:09,193 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b9d344a6097fb5c7c41dc8319fac898/C of 2b9d344a6097fb5c7c41dc8319fac898 into d609230f372b4d9b86575d6f75a40873(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:09,193 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:09,193 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898., storeName=2b9d344a6097fb5c7c41dc8319fac898/C, priority=13, startTime=1680288908752; duration=0sec 2023-03-31 18:55:09,193 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:16,970 DEBUG [Thread-1521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:63858 2023-03-31 18:55:16,970 DEBUG [Thread-1521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:17,057 DEBUG [Thread-1525] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:63858 2023-03-31 18:55:17,057 DEBUG [Thread-1525] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 484 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 5408 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 5251 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 5311 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 5415 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 5256 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 18:55:17,057 INFO [Listener at localhost/43327] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 18:55:17,057 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:63858 2023-03-31 18:55:17,058 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:17,058 INFO [Listener at localhost/43327] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 18:55:17,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 18:55:17,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:17,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 18:55:17,061 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288917061"}]},"ts":"1680288917061"} 2023-03-31 18:55:17,062 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 18:55:17,064 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 18:55:17,065 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b9d344a6097fb5c7c41dc8319fac898, UNASSIGN}] 2023-03-31 18:55:17,066 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b9d344a6097fb5c7c41dc8319fac898, UNASSIGN 2023-03-31 18:55:17,067 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=2b9d344a6097fb5c7c41dc8319fac898, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:17,067 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288917067"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288917067"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288917067"}]},"ts":"1680288917067"} 2023-03-31 18:55:17,067 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:55:17,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 18:55:17,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:17,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2b9d344a6097fb5c7c41dc8319fac898, disabling compactions & flushes 2023-03-31 18:55:17,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:17,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:17,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. after waiting 0 ms 2023-03-31 18:55:17,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:17,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2b9d344a6097fb5c7c41dc8319fac898 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-31 18:55:17,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=A 2023-03-31 18:55:17,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:17,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=B 2023-03-31 18:55:17,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:17,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b9d344a6097fb5c7c41dc8319fac898, store=C 2023-03-31 18:55:17,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:17,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 18:55:17,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/e71dc1e51b16492196c3603e0cb4bd23 2023-03-31 18:55:17,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 18:55:18,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/7e05e347642048938c1c819e95ccb384 2023-03-31 18:55:18,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 18:55:18,445 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/3821eededfb1406bb9c88aaaa5cb9d67 2023-03-31 18:55:18,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/A/e71dc1e51b16492196c3603e0cb4bd23 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e71dc1e51b16492196c3603e0cb4bd23 2023-03-31 18:55:18,452 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e71dc1e51b16492196c3603e0cb4bd23, entries=100, sequenceid=756, filesize=9.5 K 2023-03-31 18:55:18,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/B/7e05e347642048938c1c819e95ccb384 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/7e05e347642048938c1c819e95ccb384 2023-03-31 18:55:18,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/7e05e347642048938c1c819e95ccb384, entries=100, sequenceid=756, filesize=9.5 K 2023-03-31 18:55:18,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/.tmp/C/3821eededfb1406bb9c88aaaa5cb9d67 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3821eededfb1406bb9c88aaaa5cb9d67 2023-03-31 18:55:18,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3821eededfb1406bb9c88aaaa5cb9d67, entries=100, sequenceid=756, filesize=9.5 K 2023-03-31 18:55:18,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 2b9d344a6097fb5c7c41dc8319fac898 in 1240ms, sequenceid=756, compaction requested=false 2023-03-31 18:55:18,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/3f2adb2a28834915bf733ab6255b7b8c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/bdf74fa30c0346a2a5d7ca41e781ccb7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a8f4758df97c42b5b4909d8207f1a605, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/102467ed292f418eb8f8b6db5fa39c7e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/6648d603818b447c8c249d8dd33e6807, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/86b0448d4da140058a9bc13c3a125d0d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9fe90f870d7d4181976dea3fd84b7ba2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d870bbdd2b6647d0aadb75ac0a62449a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/46fcedcb3e99493395a0caa2d2567ccc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1314010fe6b74588874d44dbef07b0e4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/8af81cd69cbf48f88659c5895069baef, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/f181863dd157479da725714d7f494640, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fe0b653a52484e4aba576154b297e44f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/791e62679b6a4fba82d2a2dc2ec5f01e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/599bccb5417f4f848f2d4074aba64901, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/914612b56b35466793ae61dbe64984d2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e13806f21d6b4f62b01212214f3afef9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e21c95bcf0ae4dd18e780274c3490e35, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/74f2029adf834ec7bfc357623c98d86f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fc41c3582a7143f0911808d57c24aa6d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e4d3ace0400449599cfef3d0b618d5d8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/eca23551adbd4eeda57e6ef33c4ad9c9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a569eb6ad40c4b7fa6888d5f036549b5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d87e81b3bed44974a88c0c97bc66102b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/43797a8b25dd4deeb2394d244b5e7d29, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d2bbff5e64d44c659a61bb8cb8848ff4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/2288b59afe0c4a8ca8e7755713b4206a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9065287e06a04ff7b7c593cb9b4f2931, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e6fa9312be564fd9a401fcb922514a36, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a73e161c140141dcb7d4987637d1c14d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9410292f8b794906bee25553cd8913fa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/70e228ca05a94e2eb7834be544d18871, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1bf694edeec14a72afd3e48c73f6dd58, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/52ae0f8bf11844dc8170b97cb3bc3f21] to archive 2023-03-31 18:55:18,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:55:18,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/3f2adb2a28834915bf733ab6255b7b8c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/3f2adb2a28834915bf733ab6255b7b8c 2023-03-31 18:55:18,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/bdf74fa30c0346a2a5d7ca41e781ccb7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/bdf74fa30c0346a2a5d7ca41e781ccb7 2023-03-31 18:55:18,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a8f4758df97c42b5b4909d8207f1a605 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a8f4758df97c42b5b4909d8207f1a605 2023-03-31 18:55:18,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/102467ed292f418eb8f8b6db5fa39c7e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/102467ed292f418eb8f8b6db5fa39c7e 2023-03-31 18:55:18,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/6648d603818b447c8c249d8dd33e6807 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/6648d603818b447c8c249d8dd33e6807 2023-03-31 18:55:18,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/86b0448d4da140058a9bc13c3a125d0d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/86b0448d4da140058a9bc13c3a125d0d 2023-03-31 18:55:18,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9fe90f870d7d4181976dea3fd84b7ba2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9fe90f870d7d4181976dea3fd84b7ba2 2023-03-31 18:55:18,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d870bbdd2b6647d0aadb75ac0a62449a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d870bbdd2b6647d0aadb75ac0a62449a 2023-03-31 18:55:18,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/46fcedcb3e99493395a0caa2d2567ccc to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/46fcedcb3e99493395a0caa2d2567ccc 2023-03-31 18:55:18,490 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1314010fe6b74588874d44dbef07b0e4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1314010fe6b74588874d44dbef07b0e4 2023-03-31 18:55:18,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/8af81cd69cbf48f88659c5895069baef to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/8af81cd69cbf48f88659c5895069baef 2023-03-31 18:55:18,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/f181863dd157479da725714d7f494640 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/f181863dd157479da725714d7f494640 2023-03-31 18:55:18,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fe0b653a52484e4aba576154b297e44f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fe0b653a52484e4aba576154b297e44f 2023-03-31 18:55:18,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/791e62679b6a4fba82d2a2dc2ec5f01e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/791e62679b6a4fba82d2a2dc2ec5f01e 2023-03-31 18:55:18,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/599bccb5417f4f848f2d4074aba64901 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/599bccb5417f4f848f2d4074aba64901 2023-03-31 18:55:18,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/914612b56b35466793ae61dbe64984d2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/914612b56b35466793ae61dbe64984d2 2023-03-31 18:55:18,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e13806f21d6b4f62b01212214f3afef9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e13806f21d6b4f62b01212214f3afef9 2023-03-31 18:55:18,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e21c95bcf0ae4dd18e780274c3490e35 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e21c95bcf0ae4dd18e780274c3490e35 2023-03-31 18:55:18,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/74f2029adf834ec7bfc357623c98d86f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/74f2029adf834ec7bfc357623c98d86f 2023-03-31 18:55:18,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fc41c3582a7143f0911808d57c24aa6d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/fc41c3582a7143f0911808d57c24aa6d 2023-03-31 18:55:18,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e4d3ace0400449599cfef3d0b618d5d8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e4d3ace0400449599cfef3d0b618d5d8 2023-03-31 18:55:18,500 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/eca23551adbd4eeda57e6ef33c4ad9c9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/eca23551adbd4eeda57e6ef33c4ad9c9 2023-03-31 18:55:18,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a569eb6ad40c4b7fa6888d5f036549b5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a569eb6ad40c4b7fa6888d5f036549b5 2023-03-31 18:55:18,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d87e81b3bed44974a88c0c97bc66102b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d87e81b3bed44974a88c0c97bc66102b 2023-03-31 18:55:18,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/43797a8b25dd4deeb2394d244b5e7d29 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/43797a8b25dd4deeb2394d244b5e7d29 2023-03-31 18:55:18,503 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d2bbff5e64d44c659a61bb8cb8848ff4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/d2bbff5e64d44c659a61bb8cb8848ff4 2023-03-31 18:55:18,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/2288b59afe0c4a8ca8e7755713b4206a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/2288b59afe0c4a8ca8e7755713b4206a 2023-03-31 18:55:18,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9065287e06a04ff7b7c593cb9b4f2931 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9065287e06a04ff7b7c593cb9b4f2931 2023-03-31 18:55:18,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e6fa9312be564fd9a401fcb922514a36 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e6fa9312be564fd9a401fcb922514a36 2023-03-31 18:55:18,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a73e161c140141dcb7d4987637d1c14d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/a73e161c140141dcb7d4987637d1c14d 2023-03-31 18:55:18,507 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9410292f8b794906bee25553cd8913fa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/9410292f8b794906bee25553cd8913fa 2023-03-31 18:55:18,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/70e228ca05a94e2eb7834be544d18871 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/70e228ca05a94e2eb7834be544d18871 2023-03-31 18:55:18,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1bf694edeec14a72afd3e48c73f6dd58 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/1bf694edeec14a72afd3e48c73f6dd58 2023-03-31 18:55:18,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/52ae0f8bf11844dc8170b97cb3bc3f21 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/52ae0f8bf11844dc8170b97cb3bc3f21 2023-03-31 18:55:18,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/67755d4550224652ba8170476a90d351, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2f7f8e2c11da4f638e7ec327b7d18a5d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b2295ef6bf4b4569a3c441a17e75edcf, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/6678582d3531409bbd189ff0213715c7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/4a14cbc80e18486f902d586cbcc75024, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/e97a507f776346e59e51e76ce313be00, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/c4914d23c199415dab2c7d9748f75332, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ab380458a7c46a5b6b69e036c14f052, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/224980c1da734dfa928be8ca0af992f8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/477457a6ffe942aab23f50934d6f2c09, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19ebf44552ed4de2afdbbd28dd73ccb5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/d55ce6da052d4854973daa67ed20bc52, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10d5a4133d3b4ba1ab051e6a0070d364, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/150b88c9c0c44e19ba96e18928565a31, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2cb0c2205c8141829bd4e9e9cf94caf5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19edb54ba7894a8e8185746d94092aa8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/de7b6a17d29e40ddb36709043e5057b9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/698a26e16cd84077b54fdd6933264995, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/db1e884a99c64f458a4ebf18a6ad7854, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/048be2d608434ed4a43fe67b728b446c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/22090c38baff443a96b6a1e5d4150422, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10bdc80ca06a4220aabbfc8dd654b64e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ea29544a8ac401ea7d6d68c5739cbb7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/09d0cab94d6b46b3a54fdeb13a716ddb, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/bd299f8955644d55bd233a4b565c4895, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3b59000bb4f74c19a914c96d0d2233aa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/aadb64b8c8bc4a848cfc69e9751d2be7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/71210211238b4a48815d0e67c656b0f2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/684b22d3edd44ac3bd22a5a68f3af969, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/46162eba2ece4ed9b70ba8a57e6bf107, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10cbf25c61dd46ffbf674f1d106ce8a6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3211db8e9140454a911524a8f375c878, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/feb4d5ea0434423fb5d30bcb99cc96b1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b0e5ae2e8a7f4a04a442a72e29e13836] to archive 2023-03-31 18:55:18,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:55:18,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/67755d4550224652ba8170476a90d351 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/67755d4550224652ba8170476a90d351 2023-03-31 18:55:18,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2f7f8e2c11da4f638e7ec327b7d18a5d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2f7f8e2c11da4f638e7ec327b7d18a5d 2023-03-31 18:55:18,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b2295ef6bf4b4569a3c441a17e75edcf to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b2295ef6bf4b4569a3c441a17e75edcf 2023-03-31 18:55:18,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/6678582d3531409bbd189ff0213715c7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/6678582d3531409bbd189ff0213715c7 2023-03-31 18:55:18,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/4a14cbc80e18486f902d586cbcc75024 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/4a14cbc80e18486f902d586cbcc75024 2023-03-31 18:55:18,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/e97a507f776346e59e51e76ce313be00 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/e97a507f776346e59e51e76ce313be00 2023-03-31 18:55:18,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/c4914d23c199415dab2c7d9748f75332 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/c4914d23c199415dab2c7d9748f75332 2023-03-31 18:55:18,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ab380458a7c46a5b6b69e036c14f052 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ab380458a7c46a5b6b69e036c14f052 2023-03-31 18:55:18,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/224980c1da734dfa928be8ca0af992f8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/224980c1da734dfa928be8ca0af992f8 2023-03-31 18:55:18,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/477457a6ffe942aab23f50934d6f2c09 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/477457a6ffe942aab23f50934d6f2c09 2023-03-31 18:55:18,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19ebf44552ed4de2afdbbd28dd73ccb5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19ebf44552ed4de2afdbbd28dd73ccb5 2023-03-31 18:55:18,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/d55ce6da052d4854973daa67ed20bc52 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/d55ce6da052d4854973daa67ed20bc52 2023-03-31 18:55:18,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10d5a4133d3b4ba1ab051e6a0070d364 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10d5a4133d3b4ba1ab051e6a0070d364 2023-03-31 18:55:18,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/150b88c9c0c44e19ba96e18928565a31 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/150b88c9c0c44e19ba96e18928565a31 2023-03-31 18:55:18,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2cb0c2205c8141829bd4e9e9cf94caf5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/2cb0c2205c8141829bd4e9e9cf94caf5 2023-03-31 18:55:18,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19edb54ba7894a8e8185746d94092aa8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/19edb54ba7894a8e8185746d94092aa8 2023-03-31 18:55:18,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/de7b6a17d29e40ddb36709043e5057b9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/de7b6a17d29e40ddb36709043e5057b9 2023-03-31 18:55:18,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/698a26e16cd84077b54fdd6933264995 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/698a26e16cd84077b54fdd6933264995 2023-03-31 18:55:18,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/db1e884a99c64f458a4ebf18a6ad7854 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/db1e884a99c64f458a4ebf18a6ad7854 2023-03-31 18:55:18,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/048be2d608434ed4a43fe67b728b446c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/048be2d608434ed4a43fe67b728b446c 2023-03-31 18:55:18,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/22090c38baff443a96b6a1e5d4150422 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/22090c38baff443a96b6a1e5d4150422 2023-03-31 18:55:18,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10bdc80ca06a4220aabbfc8dd654b64e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10bdc80ca06a4220aabbfc8dd654b64e 2023-03-31 18:55:18,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ea29544a8ac401ea7d6d68c5739cbb7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/0ea29544a8ac401ea7d6d68c5739cbb7 2023-03-31 18:55:18,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/09d0cab94d6b46b3a54fdeb13a716ddb to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/09d0cab94d6b46b3a54fdeb13a716ddb 2023-03-31 18:55:18,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/bd299f8955644d55bd233a4b565c4895 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/bd299f8955644d55bd233a4b565c4895 2023-03-31 18:55:18,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3b59000bb4f74c19a914c96d0d2233aa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3b59000bb4f74c19a914c96d0d2233aa 2023-03-31 18:55:18,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/aadb64b8c8bc4a848cfc69e9751d2be7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/aadb64b8c8bc4a848cfc69e9751d2be7 2023-03-31 18:55:18,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/71210211238b4a48815d0e67c656b0f2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/71210211238b4a48815d0e67c656b0f2 2023-03-31 18:55:18,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/684b22d3edd44ac3bd22a5a68f3af969 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/684b22d3edd44ac3bd22a5a68f3af969 2023-03-31 18:55:18,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/46162eba2ece4ed9b70ba8a57e6bf107 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/46162eba2ece4ed9b70ba8a57e6bf107 2023-03-31 18:55:18,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10cbf25c61dd46ffbf674f1d106ce8a6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/10cbf25c61dd46ffbf674f1d106ce8a6 2023-03-31 18:55:18,540 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3211db8e9140454a911524a8f375c878 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/3211db8e9140454a911524a8f375c878 2023-03-31 18:55:18,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/feb4d5ea0434423fb5d30bcb99cc96b1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/feb4d5ea0434423fb5d30bcb99cc96b1 2023-03-31 18:55:18,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b0e5ae2e8a7f4a04a442a72e29e13836 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/b0e5ae2e8a7f4a04a442a72e29e13836 2023-03-31 18:55:18,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/389cb735f82a4ef7bc6e56c9586ffb4c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d7a8726ddc244f6dbdf5a4c01c6f0901, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/291a9634ca0d4914b19ee11ae56c442e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/112e2a8201cc4b92beaaa22006b3b781, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/43550e95942d42ab8f27c3b05f4bc669, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/cc3922d4b3e141c6a4d7b984709c2117, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/29daf27829e345538860ce84f6ed5f11, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3b5db18cac3e4ff78eeebcf4a854b197, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c2211646d9414331b46ec4d6a406967f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/553454c5af034bf9bb48538a13fa6f96, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3f127b6976cf4942aa8d68ac0e5468b2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/e41912e44c214deeaa3caa026f95f91b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d143781f1367497794737b01d88efb67, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/bbf9cd8811584db1933950b41f2a08b7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/f4b1fb97c625488f8d97463d352c89b1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c50e5610640b49a18fdec1f003e177e4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/421161df695340788772c14c10d5f67a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/add6769a58ea4b0983a2f94d888f0a13, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/510a4dbe791c4b3c836172ce4334eb01, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4534ce77dc3049f796354df01f13a8e0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/0777c825d49b45e89482b5442dd8110d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4e4b5b9962d44d1d80c4fb2a1ebfead9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3ae604185fce4e75a53f96674bb865db, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/142425df736246c88c14d52a115292f3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/39f25991b2ed4919a34b774a85edd4f2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/fa5e3bb9c4cd42d0bb8df342ad9c5f3e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/5e0765517f574d339423420665be730a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/714d6b95f39842ac910930861a34ac53, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/31444617951f42caa672ca134fa4214c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d746d701dd734747927a713945218b69, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/1b41777dc8204b4094bb91edfb29454a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3baa179f60e94885ab0b9425891f901f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/a0bd5cd77f1e41928b34150274373a26, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/60ac70aeae6644bead87b7ede97e303b] to archive 2023-03-31 18:55:18,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:55:18,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/389cb735f82a4ef7bc6e56c9586ffb4c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/389cb735f82a4ef7bc6e56c9586ffb4c 2023-03-31 18:55:18,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d7a8726ddc244f6dbdf5a4c01c6f0901 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d7a8726ddc244f6dbdf5a4c01c6f0901 2023-03-31 18:55:18,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/291a9634ca0d4914b19ee11ae56c442e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/291a9634ca0d4914b19ee11ae56c442e 2023-03-31 18:55:18,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/112e2a8201cc4b92beaaa22006b3b781 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/112e2a8201cc4b92beaaa22006b3b781 2023-03-31 18:55:18,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/43550e95942d42ab8f27c3b05f4bc669 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/43550e95942d42ab8f27c3b05f4bc669 2023-03-31 18:55:18,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/cc3922d4b3e141c6a4d7b984709c2117 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/cc3922d4b3e141c6a4d7b984709c2117 2023-03-31 18:55:18,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/29daf27829e345538860ce84f6ed5f11 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/29daf27829e345538860ce84f6ed5f11 2023-03-31 18:55:18,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3b5db18cac3e4ff78eeebcf4a854b197 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3b5db18cac3e4ff78eeebcf4a854b197 2023-03-31 18:55:18,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c2211646d9414331b46ec4d6a406967f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c2211646d9414331b46ec4d6a406967f 2023-03-31 18:55:18,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/553454c5af034bf9bb48538a13fa6f96 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/553454c5af034bf9bb48538a13fa6f96 2023-03-31 18:55:18,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3f127b6976cf4942aa8d68ac0e5468b2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3f127b6976cf4942aa8d68ac0e5468b2 2023-03-31 18:55:18,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/e41912e44c214deeaa3caa026f95f91b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/e41912e44c214deeaa3caa026f95f91b 2023-03-31 18:55:18,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d143781f1367497794737b01d88efb67 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d143781f1367497794737b01d88efb67 2023-03-31 18:55:18,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/bbf9cd8811584db1933950b41f2a08b7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/bbf9cd8811584db1933950b41f2a08b7 2023-03-31 18:55:18,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/f4b1fb97c625488f8d97463d352c89b1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/f4b1fb97c625488f8d97463d352c89b1 2023-03-31 18:55:18,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c50e5610640b49a18fdec1f003e177e4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/c50e5610640b49a18fdec1f003e177e4 2023-03-31 18:55:18,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/421161df695340788772c14c10d5f67a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/421161df695340788772c14c10d5f67a 2023-03-31 18:55:18,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/add6769a58ea4b0983a2f94d888f0a13 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/add6769a58ea4b0983a2f94d888f0a13 2023-03-31 18:55:18,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/510a4dbe791c4b3c836172ce4334eb01 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/510a4dbe791c4b3c836172ce4334eb01 2023-03-31 18:55:18,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4534ce77dc3049f796354df01f13a8e0 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4534ce77dc3049f796354df01f13a8e0 2023-03-31 18:55:18,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/0777c825d49b45e89482b5442dd8110d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/0777c825d49b45e89482b5442dd8110d 2023-03-31 18:55:18,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4e4b5b9962d44d1d80c4fb2a1ebfead9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/4e4b5b9962d44d1d80c4fb2a1ebfead9 2023-03-31 18:55:18,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3ae604185fce4e75a53f96674bb865db to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3ae604185fce4e75a53f96674bb865db 2023-03-31 18:55:18,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/142425df736246c88c14d52a115292f3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/142425df736246c88c14d52a115292f3 2023-03-31 18:55:18,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/39f25991b2ed4919a34b774a85edd4f2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/39f25991b2ed4919a34b774a85edd4f2 2023-03-31 18:55:18,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/fa5e3bb9c4cd42d0bb8df342ad9c5f3e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/fa5e3bb9c4cd42d0bb8df342ad9c5f3e 2023-03-31 18:55:18,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/5e0765517f574d339423420665be730a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/5e0765517f574d339423420665be730a 2023-03-31 18:55:18,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/714d6b95f39842ac910930861a34ac53 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/714d6b95f39842ac910930861a34ac53 2023-03-31 18:55:18,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/31444617951f42caa672ca134fa4214c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/31444617951f42caa672ca134fa4214c 2023-03-31 18:55:18,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d746d701dd734747927a713945218b69 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d746d701dd734747927a713945218b69 2023-03-31 18:55:18,570 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/1b41777dc8204b4094bb91edfb29454a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/1b41777dc8204b4094bb91edfb29454a 2023-03-31 18:55:18,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3baa179f60e94885ab0b9425891f901f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3baa179f60e94885ab0b9425891f901f 2023-03-31 18:55:18,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/a0bd5cd77f1e41928b34150274373a26 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/a0bd5cd77f1e41928b34150274373a26 2023-03-31 18:55:18,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/60ac70aeae6644bead87b7ede97e303b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/60ac70aeae6644bead87b7ede97e303b 2023-03-31 18:55:18,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/recovered.edits/759.seqid, newMaxSeqId=759, maxSeqId=1 2023-03-31 18:55:18,577 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898. 2023-03-31 18:55:18,577 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2b9d344a6097fb5c7c41dc8319fac898: 2023-03-31 18:55:18,578 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:18,578 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=2b9d344a6097fb5c7c41dc8319fac898, regionState=CLOSED 2023-03-31 18:55:18,578 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288918578"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288918578"}]},"ts":"1680288918578"} 2023-03-31 18:55:18,580 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-31 18:55:18,580 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 2b9d344a6097fb5c7c41dc8319fac898, server=jenkins-hbase4.apache.org,34951,1680288816339 in 1.5120 sec 2023-03-31 18:55:18,581 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-31 18:55:18,581 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b9d344a6097fb5c7c41dc8319fac898, UNASSIGN in 1.5150 sec 2023-03-31 18:55:18,581 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288918581"}]},"ts":"1680288918581"} 2023-03-31 18:55:18,582 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 18:55:18,586 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 18:55:18,587 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5280 sec 2023-03-31 18:55:19,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 18:55:19,165 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-31 18:55:19,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 18:55:19,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:19,167 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:19,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-31 18:55:19,168 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:19,172 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:19,173 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/recovered.edits] 2023-03-31 18:55:19,175 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/5fe2a4c02a684f119ab5663e44a404bf to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/5fe2a4c02a684f119ab5663e44a404bf 2023-03-31 18:55:19,176 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e71dc1e51b16492196c3603e0cb4bd23 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/A/e71dc1e51b16492196c3603e0cb4bd23 2023-03-31 18:55:19,178 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/409939c0383a43969920ec876a771df3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/409939c0383a43969920ec876a771df3 2023-03-31 18:55:19,179 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/7e05e347642048938c1c819e95ccb384 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/B/7e05e347642048938c1c819e95ccb384 2023-03-31 18:55:19,181 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3821eededfb1406bb9c88aaaa5cb9d67 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/3821eededfb1406bb9c88aaaa5cb9d67 2023-03-31 18:55:19,182 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d609230f372b4d9b86575d6f75a40873 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/C/d609230f372b4d9b86575d6f75a40873 2023-03-31 18:55:19,184 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/recovered.edits/759.seqid to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898/recovered.edits/759.seqid 2023-03-31 18:55:19,185 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/2b9d344a6097fb5c7c41dc8319fac898 2023-03-31 18:55:19,185 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:55:19,187 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:19,188 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 18:55:19,189 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 18:55:19,190 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:19,190 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 18:55:19,190 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680288919190"}]},"ts":"9223372036854775807"} 2023-03-31 18:55:19,191 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 18:55:19,191 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2b9d344a6097fb5c7c41dc8319fac898, NAME => 'TestAcidGuarantees,,1680288887029.2b9d344a6097fb5c7c41dc8319fac898.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 18:55:19,191 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 18:55:19,191 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680288919191"}]},"ts":"9223372036854775807"} 2023-03-31 18:55:19,192 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 18:55:19,194 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:19,194 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 29 msec 2023-03-31 18:55:19,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-31 18:55:19,269 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-31 18:55:19,280 INFO [Listener at localhost/43327] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 224), OpenFileDescriptor=566 (was 515) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=308 (was 351), ProcessCount=169 (was 169), AvailableMemoryMB=3461 (was 3639) 2023-03-31 18:55:19,290 INFO [Listener at localhost/43327] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=566, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=169, AvailableMemoryMB=3460 2023-03-31 18:55:19,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 18:55:19,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 18:55:19,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:19,294 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 18:55:19,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-31 18:55:19,295 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 18:55:19,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 18:55:19,296 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:19,296 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e empty. 2023-03-31 18:55:19,296 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:19,296 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:55:19,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 18:55:19,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 18:55:19,703 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 18:55:19,705 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c2adb1ceb55b533a282bcfb4ded6271e, NAME => 'TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp 2023-03-31 18:55:19,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 18:55:20,110 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:55:20,111 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c2adb1ceb55b533a282bcfb4ded6271e, disabling compactions & flushes 2023-03-31 18:55:20,111 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:20,111 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:20,111 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. after waiting 0 ms 2023-03-31 18:55:20,111 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:20,111 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:20,111 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:20,113 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 18:55:20,114 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288920113"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288920113"}]},"ts":"1680288920113"} 2023-03-31 18:55:20,114 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 18:55:20,115 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 18:55:20,115 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288920115"}]},"ts":"1680288920115"} 2023-03-31 18:55:20,116 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 18:55:20,119 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, ASSIGN}] 2023-03-31 18:55:20,120 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, ASSIGN 2023-03-31 18:55:20,121 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34951,1680288816339; forceNewPlan=false, retain=false 2023-03-31 18:55:20,272 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=c2adb1ceb55b533a282bcfb4ded6271e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:20,272 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288920272"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288920272"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288920272"}]},"ts":"1680288920272"} 2023-03-31 18:55:20,273 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:55:20,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 18:55:20,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:20,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c2adb1ceb55b533a282bcfb4ded6271e, NAME => 'TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:55:20,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:20,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:55:20,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:20,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:20,429 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:20,430 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A 2023-03-31 18:55:20,430 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A 2023-03-31 18:55:20,431 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:55:20,431 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2adb1ceb55b533a282bcfb4ded6271e columnFamilyName A 2023-03-31 18:55:20,431 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(310): Store=c2adb1ceb55b533a282bcfb4ded6271e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:55:20,431 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:20,432 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B 2023-03-31 18:55:20,432 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B 2023-03-31 18:55:20,432 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:55:20,433 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2adb1ceb55b533a282bcfb4ded6271e columnFamilyName B 2023-03-31 18:55:20,433 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(310): Store=c2adb1ceb55b533a282bcfb4ded6271e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:55:20,433 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:20,434 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C 2023-03-31 18:55:20,434 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C 2023-03-31 18:55:20,434 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:55:20,434 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2adb1ceb55b533a282bcfb4ded6271e columnFamilyName C 2023-03-31 18:55:20,435 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(310): Store=c2adb1ceb55b533a282bcfb4ded6271e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:55:20,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:20,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:20,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:20,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:55:20,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:20,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 18:55:20,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c2adb1ceb55b533a282bcfb4ded6271e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59790231, jitterRate=-0.10905613005161285}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:55:20,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:20,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., pid=37, masterSystemTime=1680288920425 2023-03-31 18:55:20,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:20,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:20,442 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=c2adb1ceb55b533a282bcfb4ded6271e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:20,442 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288920442"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680288920442"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680288920442"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680288920442"}]},"ts":"1680288920442"} 2023-03-31 18:55:20,444 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-31 18:55:20,444 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 in 170 msec 2023-03-31 18:55:20,445 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-31 18:55:20,445 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, ASSIGN in 325 msec 2023-03-31 18:55:20,445 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 18:55:20,446 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288920445"}]},"ts":"1680288920445"} 2023-03-31 18:55:20,446 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 18:55:20,448 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 18:55:20,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1560 sec 2023-03-31 18:55:21,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 18:55:21,399 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-31 18:55:21,399 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x6434ce29 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:21,405 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41507d16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:21,406 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:21,408 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47832, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:21,409 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 18:55:21,410 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39432, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 18:55:21,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 18:55:21,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 18:55:21,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:21,822 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 18:55:21,823 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 18:55:21,823 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 18:55:21,824 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-31 18:55:21,827 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, REOPEN/MOVE}] 2023-03-31 18:55:21,828 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, REOPEN/MOVE 2023-03-31 18:55:21,828 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c2adb1ceb55b533a282bcfb4ded6271e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:21,828 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288921828"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288921828"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288921828"}]},"ts":"1680288921828"} 2023-03-31 18:55:21,829 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:55:21,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:21,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c2adb1ceb55b533a282bcfb4ded6271e, disabling compactions & flushes 2023-03-31 18:55:21,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:21,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:21,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. after waiting 0 ms 2023-03-31 18:55:21,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:21,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-31 18:55:21,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:21,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:21,986 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: c2adb1ceb55b533a282bcfb4ded6271e to self. 2023-03-31 18:55:21,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:21,987 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c2adb1ceb55b533a282bcfb4ded6271e, regionState=CLOSED 2023-03-31 18:55:21,987 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288921987"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288921987"}]},"ts":"1680288921987"} 2023-03-31 18:55:21,989 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-31 18:55:21,989 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 in 159 msec 2023-03-31 18:55:21,989 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34951,1680288816339; forceNewPlan=false, retain=true 2023-03-31 18:55:22,140 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c2adb1ceb55b533a282bcfb4ded6271e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,140 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288922140"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288922140"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288922140"}]},"ts":"1680288922140"} 2023-03-31 18:55:22,141 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:55:22,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:22,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c2adb1ceb55b533a282bcfb4ded6271e, NAME => 'TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:55:22,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:22,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:55:22,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:22,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:22,298 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:22,298 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A 2023-03-31 18:55:22,298 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A 2023-03-31 18:55:22,299 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:55:22,299 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2adb1ceb55b533a282bcfb4ded6271e columnFamilyName A 2023-03-31 18:55:22,300 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(310): Store=c2adb1ceb55b533a282bcfb4ded6271e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:55:22,301 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:22,301 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B 2023-03-31 18:55:22,301 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B 2023-03-31 18:55:22,301 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:55:22,302 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2adb1ceb55b533a282bcfb4ded6271e columnFamilyName B 2023-03-31 18:55:22,302 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(310): Store=c2adb1ceb55b533a282bcfb4ded6271e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:55:22,302 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:22,303 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C 2023-03-31 18:55:22,303 DEBUG [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C 2023-03-31 18:55:22,303 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:55:22,303 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2adb1ceb55b533a282bcfb4ded6271e columnFamilyName C 2023-03-31 18:55:22,304 INFO [StoreOpener-c2adb1ceb55b533a282bcfb4ded6271e-1] regionserver.HStore(310): Store=c2adb1ceb55b533a282bcfb4ded6271e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:55:22,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:22,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:22,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:22,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:55:22,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:22,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c2adb1ceb55b533a282bcfb4ded6271e; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65008303, jitterRate=-0.03130079805850983}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:55:22,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:22,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., pid=42, masterSystemTime=1680288922293 2023-03-31 18:55:22,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:22,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:22,311 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c2adb1ceb55b533a282bcfb4ded6271e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,311 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288922311"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680288922311"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680288922311"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680288922311"}]},"ts":"1680288922311"} 2023-03-31 18:55:22,313 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-31 18:55:22,313 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 in 171 msec 2023-03-31 18:55:22,314 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-31 18:55:22,314 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, REOPEN/MOVE in 486 msec 2023-03-31 18:55:22,316 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-31 18:55:22,316 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-03-31 18:55:22,317 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-03-31 18:55:22,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-31 18:55:22,319 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:22,323 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:22,323 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:22,327 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:22,327 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:22,334 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:22,335 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x69100738 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:22,339 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c6cac96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:22,339 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x6f603c27 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:22,342 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a0df34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:22,343 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:22,346 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:22,346 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:22,349 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:22,349 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:22,355 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:22,356 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x711ebaf2 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:22,358 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d4246, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:22,359 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x0a7503d4 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:22,362 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eb8f142, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:22,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:22,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:22,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:22,372 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:22,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:22,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:22,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:22,372 DEBUG [hconnection-0xbc50524-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:22,372 DEBUG [hconnection-0x183852db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:22,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:22,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:22,373 DEBUG [hconnection-0x1c76e31e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:22,373 DEBUG [hconnection-0xa4da91a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:22,373 DEBUG [hconnection-0x3578ac5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:22,373 DEBUG [hconnection-0x5ae19359-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:22,374 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,374 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:22,376 DEBUG [hconnection-0x459d6617-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:22,376 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47852, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:22,374 DEBUG [hconnection-0xabed19e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:22,373 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47844, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:22,373 DEBUG [hconnection-0x5431b8e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:22,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:22,376 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47846, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:22,376 DEBUG [hconnection-0x2d530d52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:22,377 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,378 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:22,376 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47856, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:22,380 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47870, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:22,382 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:22,383 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47904, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:22,384 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47906, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:22,384 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:22,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:22,385 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47918, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:22,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:22,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:22,389 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:22,389 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:22,389 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:22,389 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:22,389 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:22,394 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-31 18:55:22,395 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:22,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:22,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:22,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:22,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:22,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:22,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:22,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:22,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288922408, max timestamp=1680288922410Num uniques 859; 2023-03-31 18:55:22,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288922408, max timestamp=1680288922410Num uniques 859; 2023-03-31 18:55:22,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:55:22,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:55:22,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680288922408, max timestamp=1680288922410Num uniques 859; 2023-03-31 18:55:22,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 18:55:22,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680288922415, max timestamp=1680288922423Num uniques 1018; 2023-03-31 18:55:22,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16840, totalHeapSize=38512, min timestamp=1680288922415, max timestamp=1680288922423Num uniques 1068; 2023-03-31 18:55:22,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680288922415, max timestamp=1680288922423Num uniques 1009; 2023-03-31 18:55:22,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-03-31 18:55:22,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=16848, suffix heap size=132904,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=368 2023-03-31 18:55:22,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48900, new segment data size=16840, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-03-31 18:55:22,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288982433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288982432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1680288982433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288982435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21842, totalHeapSize=49848, min timestamp=1680288922427, max timestamp=1680288922433Num uniques 1227; 2023-03-31 18:55:22,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=21842, suffix heap size=155576,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=477 2023-03-31 18:55:22,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21832, totalHeapSize=49848, min timestamp=1680288922427, max timestamp=1680288922433Num uniques 1227; 2023-03-31 18:55:22,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=56182, new segment data size=21832, suffix heap size=155576,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=477 2023-03-31 18:55:22,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21834, totalHeapSize=49848, min timestamp=1680288922427, max timestamp=1680288922433Num uniques 1227; 2023-03-31 18:55:22,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56184, new segment data size=21834, suffix heap size=155576,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=477 2023-03-31 18:55:22,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47904 deadline: 1680288982461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288982536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288982536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1680288982536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288982537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47904 deadline: 1680288982562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288982741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288982743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288982744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1680288982745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:22,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47904 deadline: 1680288982768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:22,852 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331cba6944e19e5493daff42f78e63381bd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331cba6944e19e5493daff42f78e63381bd 2023-03-31 18:55:23,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:23,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288983048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:23,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288983048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:23,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288983049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:23,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1680288983049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47904 deadline: 1680288983070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,258 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=10, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/09160208d1884418a1f31863e5d1889b 2023-03-31 18:55:23,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:23,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288983550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:23,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288983551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:23,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1680288983552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:23,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288983552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:23,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47904 deadline: 1680288983573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,669 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/96b2f9675ad947bfba0a68ee85385546 2023-03-31 18:55:23,679 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/4ee0acb8a12c4f6091cea23083e31675 2023-03-31 18:55:23,684 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/09160208d1884418a1f31863e5d1889b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/09160208d1884418a1f31863e5d1889b 2023-03-31 18:55:23,688 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/09160208d1884418a1f31863e5d1889b, entries=100, sequenceid=10, filesize=18.4 K 2023-03-31 18:55:23,688 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/96b2f9675ad947bfba0a68ee85385546 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/96b2f9675ad947bfba0a68ee85385546 2023-03-31 18:55:23,693 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/96b2f9675ad947bfba0a68ee85385546, entries=100, sequenceid=10, filesize=9.3 K 2023-03-31 18:55:23,694 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/4ee0acb8a12c4f6091cea23083e31675 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/4ee0acb8a12c4f6091cea23083e31675 2023-03-31 18:55:23,700 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/4ee0acb8a12c4f6091cea23083e31675, entries=100, sequenceid=10, filesize=9.3 K 2023-03-31 18:55:23,701 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=167.72 KB/171750 for c2adb1ceb55b533a282bcfb4ded6271e in 1309ms, sequenceid=10, compaction requested=false 2023-03-31 18:55:23,701 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-31 18:55:23,701 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:23,701 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:23,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:23,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:23,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:23,704 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:23,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:23,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:23,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:23,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:23,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:23,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:23,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:23,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23c96f82[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:23,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:23,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:23,708 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:23,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:23,708 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:23,708 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:23,708 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:23,708 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,708 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:23,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:23,712 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:23,712 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:23,713 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:23,713 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:23,713 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:23,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:23,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:23,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:23,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:23,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:23,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,717 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,720 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,720 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:23,720 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:23,720 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,720 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:23,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:23,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:23,720 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:23,720 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:23,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:23,721 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:23,721 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:23,721 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:23,721 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:23,721 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:23,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:23,821 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:23,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:23,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:23,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:23,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:23,825 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:23,825 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:23,825 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:23,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:23,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:23,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:23,827 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:23,827 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:23,827 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:23,827 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:23,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:23,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:23,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:23,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:23,830 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=167.72 KB heapSize=454.78 KB 2023-03-31 18:55:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:23,848 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033154f59e56a64a4ab795c00d2b30bd26cf to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033154f59e56a64a4ab795c00d2b30bd26cf 2023-03-31 18:55:24,111 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 18:55:24,262 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=80, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/103ae98d87764153b56ecff0d261751e 2023-03-31 18:55:24,274 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/b16411266f594eedaf1ef102c67c7390 2023-03-31 18:55:24,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:24,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:24,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:24,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1680288984567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:24,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288984573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:24,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288984573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:24,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288984573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:24,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47904 deadline: 1680288984586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:24,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288984679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:24,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288984680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:24,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288984680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,687 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/86a2e154eb0b48acbbcce57fa5877d4f 2023-03-31 18:55:24,693 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/103ae98d87764153b56ecff0d261751e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/103ae98d87764153b56ecff0d261751e 2023-03-31 18:55:24,699 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/103ae98d87764153b56ecff0d261751e, entries=150, sequenceid=80, filesize=25.1 K 2023-03-31 18:55:24,700 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/b16411266f594eedaf1ef102c67c7390 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/b16411266f594eedaf1ef102c67c7390 2023-03-31 18:55:24,706 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/b16411266f594eedaf1ef102c67c7390, entries=150, sequenceid=80, filesize=11.6 K 2023-03-31 18:55:24,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/86a2e154eb0b48acbbcce57fa5877d4f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/86a2e154eb0b48acbbcce57fa5877d4f 2023-03-31 18:55:24,718 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/86a2e154eb0b48acbbcce57fa5877d4f, entries=150, sequenceid=80, filesize=11.6 K 2023-03-31 18:55:24,719 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~454.73 KB/465648, currentSize=26.84 KB/27480 for c2adb1ceb55b533a282bcfb4ded6271e in 889ms, sequenceid=80, compaction requested=false 2023-03-31 18:55:24,719 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:24,719 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:24,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:24,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:24,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:24,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:24,721 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:24,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:24,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9dfcf[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:24,723 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:24,723 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:24,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:24,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:24,725 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:24,725 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:24,725 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:24,725 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,725 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:24,726 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:24,726 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:24,726 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,726 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:24,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:24,729 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:24,729 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:24,729 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:24,729 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:24,729 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:24,731 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:24,731 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:24,731 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:24,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:24,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:24,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:24,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:24,732 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,734 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:24,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:24,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:24,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:24,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:24,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:24,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:24,736 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:24,736 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:24,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:24,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:24,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:24,737 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:24,737 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:24,837 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:24,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:24,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:24,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:24,842 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:24,842 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:24,842 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:24,842 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:24,843 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:24,843 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:24,844 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,844 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:24,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:24,845 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:24,845 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:24,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:24,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:24,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:24,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:24,846 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:24,846 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:24,847 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 18:55:24,847 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:24,847 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:24,847 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:24,847 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:24,847 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:24,847 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:24,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:24,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:24,897 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303319d74973cc56646548b223f78ffbf675b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319d74973cc56646548b223f78ffbf675b 2023-03-31 18:55:24,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288924933, max timestamp=1680288924947Num uniques 859; 2023-03-31 18:55:24,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288924933, max timestamp=1680288924947Num uniques 859; 2023-03-31 18:55:24,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288924933, max timestamp=1680288924947Num uniques 859; 2023-03-31 18:55:24,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:24,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:24,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:24,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680288924977, max timestamp=1680288924990Num uniques 1118; 2023-03-31 18:55:24,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680288924977, max timestamp=1680288924990Num uniques 1118; 2023-03-31 18:55:24,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-31 18:55:24,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-31 18:55:24,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680288924977, max timestamp=1680288924990Num uniques 1118; 2023-03-31 18:55:24,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-31 18:55:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680288925012, max timestamp=1680288925023Num uniques 1077; 2023-03-31 18:55:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680288925012, max timestamp=1680288925023Num uniques 1077; 2023-03-31 18:55:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-31 18:55:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-31 18:55:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680288925012, max timestamp=1680288925023Num uniques 1077; 2023-03-31 18:55:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-31 18:55:25,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680288925054, max timestamp=1680288925069Num uniques 1036; 2023-03-31 18:55:25,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-31 18:55:25,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680288925054, max timestamp=1680288925069Num uniques 1036; 2023-03-31 18:55:25,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680288925054, max timestamp=1680288925069Num uniques 1036; 2023-03-31 18:55:25,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-31 18:55:25,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-31 18:55:25,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680288925095, max timestamp=1680288925113Num uniques 1095; 2023-03-31 18:55:25,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-31 18:55:25,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680288925095, max timestamp=1680288925113Num uniques 1095; 2023-03-31 18:55:25,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-31 18:55:25,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680288925095, max timestamp=1680288925113Num uniques 1095; 2023-03-31 18:55:25,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-31 18:55:25,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288985151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288985158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680288925148, max timestamp=1680288925159Num uniques 1104; 2023-03-31 18:55:25,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-31 18:55:25,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680288925148, max timestamp=1680288925159Num uniques 1104; 2023-03-31 18:55:25,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1680288925141, max timestamp=1680288925159Num uniques 1104; 2023-03-31 18:55:25,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-31 18:55:25,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=16210, suffix heap size=142784,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=354 2023-03-31 18:55:25,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680288925239, max timestamp=1680288925268Num uniques 1013; 2023-03-31 18:55:25,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-31 18:55:25,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680288925239, max timestamp=1680288925268Num uniques 1213; 2023-03-31 18:55:25,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680288925239, max timestamp=1680288925269Num uniques 1013; 2023-03-31 18:55:25,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-31 18:55:25,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=55548, new segment data size=9748, suffix heap size=154120,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1213, new segment cells count=213 2023-03-31 18:55:25,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680288925295, max timestamp=1680288925305Num uniques 1072; 2023-03-31 18:55:25,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-31 18:55:25,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680288925295, max timestamp=1680288925305Num uniques 1072; 2023-03-31 18:55:25,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680288925295, max timestamp=1680288925305Num uniques 1072; 2023-03-31 18:55:25,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-31 18:55:25,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-31 18:55:25,309 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=87, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/22e75b96ed444a26adb88727bdf4b508 2023-03-31 18:55:25,318 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/dc491004760541f7904d3a284a3f1485 2023-03-31 18:55:25,326 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/23383f6922ed416dbdfebeeec8edc859 2023-03-31 18:55:25,332 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/22e75b96ed444a26adb88727bdf4b508 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/22e75b96ed444a26adb88727bdf4b508 2023-03-31 18:55:25,335 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/22e75b96ed444a26adb88727bdf4b508, entries=150, sequenceid=87, filesize=25.1 K 2023-03-31 18:55:25,336 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/dc491004760541f7904d3a284a3f1485 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/dc491004760541f7904d3a284a3f1485 2023-03-31 18:55:25,339 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/dc491004760541f7904d3a284a3f1485, entries=150, sequenceid=87, filesize=11.6 K 2023-03-31 18:55:25,343 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/23383f6922ed416dbdfebeeec8edc859 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/23383f6922ed416dbdfebeeec8edc859 2023-03-31 18:55:25,346 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/23383f6922ed416dbdfebeeec8edc859, entries=150, sequenceid=87, filesize=11.6 K 2023-03-31 18:55:25,347 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=134.18 KB/137400 for c2adb1ceb55b533a282bcfb4ded6271e in 500ms, sequenceid=87, compaction requested=true 2023-03-31 18:55:25,347 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:25,347 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:25,351 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278f6945[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:25,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:25,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:25,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=140.89 KB heapSize=399 KB 2023-03-31 18:55:25,355 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,355 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:25,355 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:25,355 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:25,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,356 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:25,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680288925330, max timestamp=1680288925353Num uniques 1081; 2023-03-31 18:55:25,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-31 18:55:25,356 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:25,356 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 18:55:25,356 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:25,356 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:25,356 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:25,356 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 18:55:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:25,359 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:25,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:25,371 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:25,371 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:25,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:25,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:25,372 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:25,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:25,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,375 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,376 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331e7a67f988f79445ebf285b7e7891f4f5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e7a67f988f79445ebf285b7e7891f4f5 2023-03-31 18:55:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,384 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288985381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:25,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:25,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:25,388 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:25,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:25,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:25,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:25,389 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:25,389 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:25,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:25,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:25,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288985384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288985385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,489 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:25,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:25,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:25,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288985488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:25,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:25,494 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:25,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:25,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:25,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:25,497 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,497 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,498 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:25,498 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:25,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:25,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:25,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:25,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:25,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:25,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:25,500 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:25,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288985494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288985495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,503 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,503 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,503 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,503 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64f40635[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:25,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:25,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,505 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,506 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:25,506 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:25,508 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:25,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:25,511 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:25,511 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:25,511 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:25,511 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:25,511 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:25,512 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,512 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,512 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:25,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,514 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:25,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,517 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:25,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:25,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:25,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:25,517 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,517 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,517 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,517 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,517 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:25,518 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:25,518 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:25,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:25,618 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:25,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:25,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:25,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:25,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:25,621 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:25,621 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:25,621 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:25,621 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:25,622 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,622 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,623 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:25,623 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:25,623 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,623 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:25,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:25,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:25,624 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:25,625 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:25,625 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:25,625 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:25,628 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,628 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:25,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca9c056[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:25,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:25,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,631 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:25,631 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,632 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:25,633 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:25,633 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:25,633 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,633 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:25,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:25,635 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:25,635 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:25,635 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:25,635 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:25,635 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:25,637 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,637 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,637 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:25,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,637 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:25,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:25,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:25,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:25,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:25,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,652 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:25,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:25,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:25,652 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:25,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,653 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,653 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,653 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,653 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:25,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:25,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288985693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288985703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:25,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288985703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,753 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:25,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:25,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:25,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:25,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:25,756 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:25,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:25,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:25,756 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:25,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,757 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,759 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:25,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:25,759 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,759 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:25,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:25,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:25,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:25,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:25,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:25,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:25,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:25,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:25,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:25,760 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:25,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:25,763 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:25,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44d641ab[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:25,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:25,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,766 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,766 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:25,766 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,766 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:25,766 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:25,766 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:25,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:25,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:25,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:25,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:25,771 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:25,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:25,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:25,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:25,771 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:25,776 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,776 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,776 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:25,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:25,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:25,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:25,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:25,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:25,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:25,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:25,783 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:25,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:25,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:25,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,783 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:25,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,783 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:25,784 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,784 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,784 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,787 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=246, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/34c59f49b9924b5b99ca1003aed63d1b 2023-03-31 18:55:25,797 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/93349635496a4a87bc090a3b29cd0142 2023-03-31 18:55:25,884 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:25,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:25,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:25,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:25,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:25,887 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:25,887 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:25,887 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:25,887 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:25,888 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,888 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,889 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,889 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:25,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:25,889 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,889 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:25,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:25,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:25,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:25,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:25,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:25,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:25,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:25,890 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:25,890 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:25,891 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:25,891 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:25,892 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,892 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,892 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,892 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:25,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@184fdd35[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:25,897 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:25,897 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,899 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:25,899 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:25,899 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:25,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:25,901 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:25,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:25,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:25,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:25,904 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:25,904 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:25,904 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:25,904 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:25,904 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:25,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:25,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:25,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,907 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:25,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:25,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:25,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:25,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:25,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:25,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:25,910 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,910 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:25,910 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:25,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:25,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:25,910 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:25,910 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:25,910 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:25,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:26,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288986002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,010 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:26,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:26,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288986011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:26,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288986012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:26,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:26,014 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:26,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:26,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:26,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:26,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,015 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:26,015 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:26,016 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,016 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:26,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:26,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:26,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:26,017 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,017 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:26,017 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:26,017 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:26,019 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,019 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:26,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4ea5a7[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:26,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:26,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:26,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:26,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:26,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:26,024 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:26,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:26,027 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:26,027 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:26,027 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:26,027 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:26,027 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:26,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:26,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,030 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:26,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:26,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:26,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:26,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:26,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:26,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:26,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,033 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:26,033 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:26,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:26,134 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:26,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:26,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:26,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:26,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:26,137 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:26,137 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:26,137 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:26,137 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:26,137 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,137 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,138 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,138 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:26,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:26,138 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,139 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:26,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:26,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:26,141 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,141 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:26,141 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:26,141 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:26,142 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:26,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e277508[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:26,144 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:26,144 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,145 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:26,146 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:26,146 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:26,146 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,146 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:26,148 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:26,150 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:26,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:26,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:26,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:26,150 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:26,151 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,151 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,151 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,152 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,154 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,156 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:26,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:26,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:26,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:26,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:26,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:26,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:26,157 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:26,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,157 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:26,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:26,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/116c9585730f4b4880e5a08438f362e8 2023-03-31 18:55:26,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/34c59f49b9924b5b99ca1003aed63d1b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/34c59f49b9924b5b99ca1003aed63d1b 2023-03-31 18:55:26,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/34c59f49b9924b5b99ca1003aed63d1b, entries=300, sequenceid=246, filesize=45.8 K 2023-03-31 18:55:26,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/93349635496a4a87bc090a3b29cd0142 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/93349635496a4a87bc090a3b29cd0142 2023-03-31 18:55:26,220 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/93349635496a4a87bc090a3b29cd0142, entries=150, sequenceid=246, filesize=11.8 K 2023-03-31 18:55:26,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/116c9585730f4b4880e5a08438f362e8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/116c9585730f4b4880e5a08438f362e8 2023-03-31 18:55:26,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/116c9585730f4b4880e5a08438f362e8, entries=150, sequenceid=246, filesize=11.8 K 2023-03-31 18:55:26,227 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~116.29 KB/119080, heapSize ~326.09 KB/333920, currentSize=67.09 KB/68700 for c2adb1ceb55b533a282bcfb4ded6271e in 873ms, sequenceid=246, compaction requested=true 2023-03-31 18:55:26,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:26,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:26,228 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:26,228 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:26,228 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:26,228 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:26,229 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117160 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:26,229 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/A is initiating minor compaction (all files) 2023-03-31 18:55:26,229 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:26,229 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/A in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,229 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/B is initiating minor compaction (all files) 2023-03-31 18:55:26,229 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/09160208d1884418a1f31863e5d1889b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/103ae98d87764153b56ecff0d261751e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/22e75b96ed444a26adb88727bdf4b508, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/34c59f49b9924b5b99ca1003aed63d1b] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=114.4 K 2023-03-31 18:55:26,229 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/B in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,229 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/96b2f9675ad947bfba0a68ee85385546, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/b16411266f594eedaf1ef102c67c7390, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/dc491004760541f7904d3a284a3f1485, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/93349635496a4a87bc090a3b29cd0142] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=44.3 K 2023-03-31 18:55:26,229 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 09160208d1884418a1f31863e5d1889b, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680288922387 2023-03-31 18:55:26,229 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 96b2f9675ad947bfba0a68ee85385546, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680288922387 2023-03-31 18:55:26,230 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 103ae98d87764153b56ecff0d261751e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680288922455 2023-03-31 18:55:26,230 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b16411266f594eedaf1ef102c67c7390, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680288922455 2023-03-31 18:55:26,230 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 22e75b96ed444a26adb88727bdf4b508, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680288924571 2023-03-31 18:55:26,230 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting dc491004760541f7904d3a284a3f1485, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680288924571 2023-03-31 18:55:26,230 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 34c59f49b9924b5b99ca1003aed63d1b, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680288925330 2023-03-31 18:55:26,230 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 93349635496a4a87bc090a3b29cd0142, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680288925338 2023-03-31 18:55:26,248 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#B#compaction#298 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:26,257 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:26,260 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#A#compaction#297 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:26,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:26,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:26,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:26,263 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:26,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:26,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:26,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:26,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,266 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:26,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:26,266 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:26,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:26,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:26,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:26,270 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/8e42cf61dbd6465095181ac043f9b58e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/8e42cf61dbd6465095181ac043f9b58e 2023-03-31 18:55:26,270 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,270 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:26,270 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 18:55:26,270 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:26,270 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:26,270 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:26,270 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:26,270 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:26,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:26,279 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/B of c2adb1ceb55b533a282bcfb4ded6271e into 8e42cf61dbd6465095181ac043f9b58e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:26,279 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:26,279 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/B, priority=12, startTime=1680288926228; duration=0sec 2023-03-31 18:55:26,279 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:26,279 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:26,280 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:26,280 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/C is initiating minor compaction (all files) 2023-03-31 18:55:26,280 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/C in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,280 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/4ee0acb8a12c4f6091cea23083e31675, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/86a2e154eb0b48acbbcce57fa5877d4f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/23383f6922ed416dbdfebeeec8edc859, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/116c9585730f4b4880e5a08438f362e8] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=44.3 K 2023-03-31 18:55:26,281 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4ee0acb8a12c4f6091cea23083e31675, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680288922387 2023-03-31 18:55:26,281 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 86a2e154eb0b48acbbcce57fa5877d4f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680288922455 2023-03-31 18:55:26,282 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 23383f6922ed416dbdfebeeec8edc859, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680288924571 2023-03-31 18:55:26,282 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 116c9585730f4b4880e5a08438f362e8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680288925338 2023-03-31 18:55:26,296 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#C#compaction#300 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:26,307 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331ff218b660519403c9af7b94fe275d230 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ff218b660519403c9af7b94fe275d230 2023-03-31 18:55:26,314 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=259, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/4c9cc339e26d4059915af9c2591e071d 2023-03-31 18:55:26,328 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/13f29aa4c52348c4b4f42bfab84d5ce5 2023-03-31 18:55:26,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:26,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:26,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288986558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288986563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288986563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1680288986579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,589 DEBUG [Thread-2254] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:26,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47904 deadline: 1680288986602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,605 DEBUG [Thread-2248] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:26,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288986664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288986675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288986675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,716 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/b19d32f21322449cb802781c964811e3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b19d32f21322449cb802781c964811e3 2023-03-31 18:55:26,720 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/C of c2adb1ceb55b533a282bcfb4ded6271e into b19d32f21322449cb802781c964811e3(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:26,720 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:26,720 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/C, priority=12, startTime=1680288926228; duration=0sec 2023-03-31 18:55:26,720 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:26,743 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/8fe4be991cb047d99e4ba58ce6edbe34 2023-03-31 18:55:26,746 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/4c9cc339e26d4059915af9c2591e071d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/4c9cc339e26d4059915af9c2591e071d 2023-03-31 18:55:26,749 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/4c9cc339e26d4059915af9c2591e071d, entries=150, sequenceid=259, filesize=25.3 K 2023-03-31 18:55:26,750 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/13f29aa4c52348c4b4f42bfab84d5ce5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/13f29aa4c52348c4b4f42bfab84d5ce5 2023-03-31 18:55:26,754 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/13f29aa4c52348c4b4f42bfab84d5ce5, entries=150, sequenceid=259, filesize=11.8 K 2023-03-31 18:55:26,755 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/8fe4be991cb047d99e4ba58ce6edbe34 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/8fe4be991cb047d99e4ba58ce6edbe34 2023-03-31 18:55:26,761 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/8fe4be991cb047d99e4ba58ce6edbe34, entries=150, sequenceid=259, filesize=11.8 K 2023-03-31 18:55:26,762 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for c2adb1ceb55b533a282bcfb4ded6271e in 492ms, sequenceid=259, compaction requested=false 2023-03-31 18:55:26,762 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:26,762 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,762 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:26,762 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:26,762 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,764 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:26,764 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:26,765 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,765 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688aa47e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:26,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:26,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,769 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:26,769 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:26,769 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:26,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:26,770 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,771 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,771 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:26,771 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:26,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:26,772 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:26,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:26,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:26,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:26,772 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:26,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:26,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:26,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,778 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,778 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,778 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,778 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:26,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:26,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:26,779 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:26,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:26,779 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:26,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:26,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:26,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:26,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:26,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:26,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-31 18:55:26,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:26,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:26,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:26,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:26,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:26,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:26,880 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:26,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:26,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:26,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:26,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:26,885 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:26,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:26,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:26,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:26,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,886 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,887 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:26,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:26,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:26,888 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,888 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,888 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:26,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:26,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:26,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:26,890 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,890 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:26,890 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:26,890 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:26,891 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:26,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:26,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eca6a2f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:26,893 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,894 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:26,895 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,895 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:26,896 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:26,898 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:26,898 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:26,898 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:26,898 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:26,898 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:26,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:26,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288986896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:26,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:26,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:26,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:26,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:26,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:26,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:26,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:26,904 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:26,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:26,904 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:26,905 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:26,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288986901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:26,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:26,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288986904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,005 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:27,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:27,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:27,009 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:27,009 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,009 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:27,009 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:27,009 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,009 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,011 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,011 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,012 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,012 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:27,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:27,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:27,013 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,013 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:27,013 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:27,013 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:27,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:27,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288987006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:27,015 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:27,017 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:27,017 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278095cc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:27,018 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:27,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:27,018 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,018 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:27,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:27,019 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,019 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,019 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:27,022 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:27,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:27,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:27,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:27,022 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:27,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:27,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288987014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:27,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288987014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,024 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,024 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,025 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:27,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:27,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,028 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:27,028 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:27,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,103 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/583034e6e8b249f39c29df398fc39fbe as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/583034e6e8b249f39c29df398fc39fbe 2023-03-31 18:55:27,106 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/A of c2adb1ceb55b533a282bcfb4ded6271e into 583034e6e8b249f39c29df398fc39fbe(size=25.4 K), total size for store is 50.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:27,106 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:27,106 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/A, priority=12, startTime=1680288926227; duration=0sec 2023-03-31 18:55:27,106 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:27,129 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:27,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:27,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:27,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:27,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:27,133 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:27,133 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,133 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:27,133 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:27,133 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,133 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,141 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,141 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,141 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,141 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:27,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:27,142 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:27,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:27,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:27,144 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aa2b01[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:27,146 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:27,146 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,147 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,147 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:27,148 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,148 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:27,150 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:27,152 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:27,153 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:27,153 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:27,153 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:27,153 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:27,154 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,154 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,154 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,154 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,157 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,159 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,159 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,159 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,159 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,159 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:27,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:27,159 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,159 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:27,159 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:27,159 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:27,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:27,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288987215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:27,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288987223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:27,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288987224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,259 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:27,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:27,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:27,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:27,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:27,262 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:27,262 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,262 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:27,262 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:27,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,263 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,264 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,264 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,264 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,264 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:27,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,269 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:27,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fbec67e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:27,271 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:27,271 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:27,272 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,272 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:27,273 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:27,275 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:27,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:27,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:27,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:27,275 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:27,277 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,277 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,277 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,277 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,279 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,281 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,281 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,281 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,281 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,281 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:27,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:27,281 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,282 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,282 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,282 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:27,282 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:27,282 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,282 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:27,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,289 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331aa0c0226dc1d4f68a95bb476319458c4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331aa0c0226dc1d4f68a95bb476319458c4 2023-03-31 18:55:27,296 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=282, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/294d331b0c184f4ebc28c6a48b312084 2023-03-31 18:55:27,382 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:27,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:27,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:27,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:27,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:27,385 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:27,386 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,386 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:27,386 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:27,386 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,386 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,388 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,388 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,388 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:27,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:27,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:27,390 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,390 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:27,390 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:27,390 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:27,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@627848b4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:27,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:27,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,395 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,395 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:27,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:27,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:27,396 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,396 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,400 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:27,403 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:27,403 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:27,403 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:27,403 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:27,403 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:27,404 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,404 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,405 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:27,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:27,409 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,409 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:27,409 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:27,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,410 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,410 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,410 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,510 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:27,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:27,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:27,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:27,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:27,513 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:27,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:27,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:27,514 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,514 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,515 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,515 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,515 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,515 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:27,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:27,516 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:27,516 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,516 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:27,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:27,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288987519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,525 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,525 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,525 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,525 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41759fc2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:27,526 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:27,526 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,527 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,528 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:27,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:27,528 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:27,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,529 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:27,531 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:27,531 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:27,531 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:27,531 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:27,531 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:27,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:27,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288987527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:27,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288987527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,534 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,534 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,534 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,534 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,538 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,538 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:27,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:27,538 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,538 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,539 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:27,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:27,539 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:27,539 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,539 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,539 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,539 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,539 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,639 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:27,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:27,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:27,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:27,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:27,642 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:27,642 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,642 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:27,642 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:27,643 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,643 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,644 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,644 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:27,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:27,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:27,646 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,646 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:27,646 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:27,646 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:27,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,647 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:27,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125446ce[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:27,649 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:27,649 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,651 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:27,651 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:27,651 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:27,651 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,651 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:27,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,652 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,652 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:27,655 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:27,655 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:27,655 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:27,655 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:27,655 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:27,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,656 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,657 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,658 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,660 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:27,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:27,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:27,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,661 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,661 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:27,661 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:27,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/1f605752eb7f41a7b164a283d14476e1 2023-03-31 18:55:27,761 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:27,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:27,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:27,764 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:27,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:27,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:27,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,765 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,767 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:27,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:27,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:27,775 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,775 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:27,775 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:27,775 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:27,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:27,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:27,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:27,776 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,776 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:27,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b10efcc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:27,778 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:27,778 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:27,780 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:27,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,780 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:27,780 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:27,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:27,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:27,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:27,783 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:27,783 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:27,783 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:27,783 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:27,783 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:27,786 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,786 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,786 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,787 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,788 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,790 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:27,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:27,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:27,791 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:27,791 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:27,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,891 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:27,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:27,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:27,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:27,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:27,900 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:27,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:27,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:27,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,902 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:27,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:27,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:27,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:27,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:27,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:27,907 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,907 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:27,907 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:27,907 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:27,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:27,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b8d31f2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:27,910 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:27,910 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,913 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:27,913 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:27,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:27,913 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:27,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:27,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:27,926 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:27,926 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:27,926 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:27,926 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:27,926 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:27,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:27,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,929 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:27,931 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,933 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:27,933 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:27,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:27,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:27,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:27,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:27,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:27,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:27,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:27,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,934 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:27,934 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:27,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:27,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:27,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:28,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:28,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47844 deadline: 1680288988027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,034 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:28,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:28,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:28,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:28,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:28,037 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:28,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:28,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:28,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:28,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,040 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:28,040 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:28,040 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,040 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288988036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:28,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:28,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:28,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:28,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:28,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:28,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288988037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:28,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:28,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:28,044 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,044 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:28,045 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:28,045 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:28,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:28,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:28,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:28,047 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:28,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16eb2ace[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:28,049 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:28,049 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:28,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:28,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:28,050 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:28,052 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,052 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:28,052 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,052 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:28,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:28,054 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:28,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:28,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:28,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:28,054 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:28,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:28,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:28,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:28,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:28,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:28,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:28,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:28,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:28,061 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:28,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,061 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:28,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,061 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:28,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:28,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/9ec3b4a0b5014d69843cbfb981331b75 2023-03-31 18:55:28,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/294d331b0c184f4ebc28c6a48b312084 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/294d331b0c184f4ebc28c6a48b312084 2023-03-31 18:55:28,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/294d331b0c184f4ebc28c6a48b312084, entries=200, sequenceid=282, filesize=32.3 K 2023-03-31 18:55:28,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/1f605752eb7f41a7b164a283d14476e1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1f605752eb7f41a7b164a283d14476e1 2023-03-31 18:55:28,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1f605752eb7f41a7b164a283d14476e1, entries=150, sequenceid=282, filesize=11.9 K 2023-03-31 18:55:28,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/9ec3b4a0b5014d69843cbfb981331b75 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ec3b4a0b5014d69843cbfb981331b75 2023-03-31 18:55:28,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ec3b4a0b5014d69843cbfb981331b75, entries=150, sequenceid=282, filesize=11.9 K 2023-03-31 18:55:28,139 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for c2adb1ceb55b533a282bcfb4ded6271e in 1263ms, sequenceid=282, compaction requested=true 2023-03-31 18:55:28,139 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:28,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:28,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:28,139 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:28,139 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:28,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:28,140 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85066 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:28,140 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/A is initiating minor compaction (all files) 2023-03-31 18:55:28,140 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36446 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:28,140 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/A in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,140 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/B is initiating minor compaction (all files) 2023-03-31 18:55:28,140 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/583034e6e8b249f39c29df398fc39fbe, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/4c9cc339e26d4059915af9c2591e071d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/294d331b0c184f4ebc28c6a48b312084] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=83.1 K 2023-03-31 18:55:28,140 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/B in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,140 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/8e42cf61dbd6465095181ac043f9b58e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/13f29aa4c52348c4b4f42bfab84d5ce5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1f605752eb7f41a7b164a283d14476e1] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=35.6 K 2023-03-31 18:55:28,141 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 8e42cf61dbd6465095181ac043f9b58e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680288925338 2023-03-31 18:55:28,141 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 583034e6e8b249f39c29df398fc39fbe, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680288925338 2023-03-31 18:55:28,141 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 13f29aa4c52348c4b4f42bfab84d5ce5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1680288925372 2023-03-31 18:55:28,141 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 4c9cc339e26d4059915af9c2591e071d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1680288925372 2023-03-31 18:55:28,142 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 1f605752eb7f41a7b164a283d14476e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680288926562 2023-03-31 18:55:28,142 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 294d331b0c184f4ebc28c6a48b312084, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680288926531 2023-03-31 18:55:28,150 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#B#compaction#307 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:28,157 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#A#compaction#306 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:28,162 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:28,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:28,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:28,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:28,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:28,171 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:28,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:28,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:28,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:28,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,173 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:28,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:28,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:28,175 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/351430e1d42b443eb06c677c5956e422 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/351430e1d42b443eb06c677c5956e422 2023-03-31 18:55:28,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:28,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:28,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:28,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:28,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:28,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:28,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:28,176 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,176 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:28,176 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:55:28,177 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:28,177 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:28,177 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:28,177 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:28,177 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:28,177 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:28,180 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/B of c2adb1ceb55b533a282bcfb4ded6271e into 351430e1d42b443eb06c677c5956e422(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:28,180 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:28,180 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/B, priority=13, startTime=1680288928139; duration=0sec 2023-03-31 18:55:28,181 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:28,181 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:28,182 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36446 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:28,182 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/C is initiating minor compaction (all files) 2023-03-31 18:55:28,182 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/C in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,182 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b19d32f21322449cb802781c964811e3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/8fe4be991cb047d99e4ba58ce6edbe34, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ec3b4a0b5014d69843cbfb981331b75] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=35.6 K 2023-03-31 18:55:28,183 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b19d32f21322449cb802781c964811e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680288925338 2023-03-31 18:55:28,183 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 8fe4be991cb047d99e4ba58ce6edbe34, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1680288925372 2023-03-31 18:55:28,183 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9ec3b4a0b5014d69843cbfb981331b75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680288926562 2023-03-31 18:55:28,193 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#C#compaction#309 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:28,213 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/a7c7b51f0a484838b6e036b110479ac7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/a7c7b51f0a484838b6e036b110479ac7 2023-03-31 18:55:28,217 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/C of c2adb1ceb55b533a282bcfb4ded6271e into a7c7b51f0a484838b6e036b110479ac7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:28,217 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:28,217 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/C, priority=13, startTime=1680288928139; duration=0sec 2023-03-31 18:55:28,218 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:28,598 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033173e48f2d8b814ee7bba044c793221539 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033173e48f2d8b814ee7bba044c793221539 2023-03-31 18:55:28,603 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=294, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/608cecfd4c3b448c9e498047d444889b 2023-03-31 18:55:28,616 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/128ec65b25a1414fb13baf85c752f479 2023-03-31 18:55:28,627 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/9ead9b4416d04105adeec04ac0fb20f2 2023-03-31 18:55:28,631 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/608cecfd4c3b448c9e498047d444889b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/608cecfd4c3b448c9e498047d444889b 2023-03-31 18:55:28,636 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/608cecfd4c3b448c9e498047d444889b, entries=150, sequenceid=294, filesize=25.4 K 2023-03-31 18:55:28,636 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/128ec65b25a1414fb13baf85c752f479 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/128ec65b25a1414fb13baf85c752f479 2023-03-31 18:55:28,640 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/128ec65b25a1414fb13baf85c752f479, entries=150, sequenceid=294, filesize=11.9 K 2023-03-31 18:55:28,641 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/9ead9b4416d04105adeec04ac0fb20f2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ead9b4416d04105adeec04ac0fb20f2 2023-03-31 18:55:28,645 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ead9b4416d04105adeec04ac0fb20f2, entries=150, sequenceid=294, filesize=11.9 K 2023-03-31 18:55:28,645 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for c2adb1ceb55b533a282bcfb4ded6271e in 469ms, sequenceid=294, compaction requested=false 2023-03-31 18:55:28,645 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:28,645 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:28,647 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b97ceb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:28,649 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:28,649 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:28,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:28,653 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:28,656 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:28,656 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:28,656 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:28,656 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:28,656 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:28,657 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,657 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:28,657 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,660 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,662 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,664 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:28,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:28,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:28,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,665 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,665 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:28,665 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:28,666 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:28,666 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:28,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:28,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:28,765 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:28,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:28,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:28,768 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:28,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:28,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:28,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:28,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:28,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:28,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:28,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:28,777 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,777 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:28,778 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:28,778 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:28,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,780 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:28,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a0b997f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:28,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:28,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,786 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:28,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:28,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:28,786 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,786 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:28,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,790 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:28,790 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,790 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:28,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:28,802 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:28,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:28,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:28,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:28,802 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:28,807 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,807 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,807 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,808 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,810 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,812 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:28,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:28,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:28,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:28,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:28,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:28,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:28,813 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,814 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:28,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,814 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:28,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,914 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:28,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:28,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:28,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:28,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:28,917 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:28,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:28,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:28,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:28,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:28,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:28,920 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,921 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,921 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:28,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:28,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:28,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:28,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:28,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:28,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:28,928 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:28,928 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:28,928 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,928 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:28,928 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:28,928 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:28,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:28,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:28,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:28,929 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:28,931 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,931 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:28,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@689b256d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:28,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:28,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,935 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:28,935 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,935 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:28,936 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:28,939 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:28,939 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:28,939 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:28,939 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:28,939 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:28,940 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,940 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:28,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,940 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:28,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,943 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,946 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:28,946 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:28,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:28,947 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,947 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,947 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:28,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:28,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:28,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:28,947 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:28,947 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:28,947 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,947 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:28,947 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:28,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:28,947 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:28,947 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:29,047 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:29,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:29,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:29,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:29,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:29,051 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:29,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:29,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:29,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,052 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,052 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:29,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:29,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:29,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:29,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:29,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:29,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:29,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:29,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:29,054 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,054 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:29,054 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:29,054 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:29,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:29,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:29,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:29,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:29,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:29,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2769cfea[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:29,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:29,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,059 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:29,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:29,062 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:29,065 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:29,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:29,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:29,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:29,065 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:29,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,067 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,068 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,070 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,070 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,070 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,070 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:29,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:29,070 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:29,071 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,071 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:29,071 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,071 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:29,071 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,071 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,071 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:29,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:55:29,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:29,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:29,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:29,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:29,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:29,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:29,135 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033152cb769ec77a44a6ad0b5a12c0fd58e0 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033152cb769ec77a44a6ad0b5a12c0fd58e0 2023-03-31 18:55:29,144 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=310, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/b3978dc77a0849d4b51c26dac8266405 2023-03-31 18:55:29,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288929142, max timestamp=1680288929161Num uniques 859; 2023-03-31 18:55:29,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:29,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288929142, max timestamp=1680288929161Num uniques 859; 2023-03-31 18:55:29,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:29,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288929142, max timestamp=1680288929161Num uniques 859; 2023-03-31 18:55:29,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:29,171 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:29,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:29,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:29,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:29,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:29,174 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:29,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:29,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:29,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,176 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,176 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,176 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,176 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:29,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:29,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:29,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:29,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:29,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:29,178 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,178 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:29,178 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:29,178 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:29,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:29,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4adab1c8[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:29,182 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:29,182 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,183 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,183 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:29,183 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:29,183 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:29,183 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,184 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:29,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:29,187 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:29,187 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:29,187 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:29,187 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:29,187 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:29,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,189 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:29,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:29,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:29,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,197 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,198 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:29,198 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,198 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288989192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288989194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288989199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,298 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:29,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:29,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:29,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:29,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:29,302 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:29,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:29,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:29,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,303 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,303 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,303 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:29,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:29,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:29,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:29,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:29,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:29,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:29,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47870 deadline: 1680288989299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47856 deadline: 1680288989300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,310 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cbc098d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:29,311 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:29,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,312 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288989306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,313 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:29,314 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,314 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:29,316 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:29,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:29,318 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:29,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:29,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:29,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:29,318 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:29,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,321 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,323 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,324 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:29,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:29,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,324 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,324 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,324 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:29,324 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:29,388 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/9ca519b5262a4a4593110db82e292c25 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ca519b5262a4a4593110db82e292c25 2023-03-31 18:55:29,392 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/A of c2adb1ceb55b533a282bcfb4ded6271e into 9ca519b5262a4a4593110db82e292c25(size=25.7 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-31 18:55:29,392 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:29,392 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/A, priority=13, startTime=1680288928139; duration=1sec 2023-03-31 18:55:29,392 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:29,424 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:29,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:29,427 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:29,427 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,428 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:29,428 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:29,428 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,428 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,429 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,429 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,430 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,430 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:29,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:29,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:29,431 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,432 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:29,432 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:29,432 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:29,434 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:29,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbdf93a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:29,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:29,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,437 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,437 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:29,437 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:29,439 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:29,439 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,439 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,439 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:29,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:29,441 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:29,441 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:29,441 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:29,441 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:29,441 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:29,443 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,443 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,443 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:29,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:29,449 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,449 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:29,449 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:29,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288989510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288989511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288989514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,550 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:29,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:29,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:29,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:29,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:29,563 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:29,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:29,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:29,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/0da9ac087af54f748750405a83b9487e 2023-03-31 18:55:29,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,564 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,564 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:29,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:29,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:29,569 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,569 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:29,569 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:29,569 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:29,571 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c26bf8[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:29,575 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:29,575 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,577 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:29,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:29,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:29,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:29,598 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:29,598 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:29,598 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:29,598 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:29,598 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:29,601 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,602 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,602 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,603 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,604 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,609 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,609 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:29,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:29,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,610 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:29,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,610 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:29,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,710 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:29,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:29,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:29,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:29,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:29,713 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:29,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:29,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:29,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,716 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:29,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:29,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:29,717 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:29,718 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,718 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:29,718 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:29,718 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:29,719 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,720 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,720 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:29,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b89d859[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:29,721 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:29,722 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,723 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:29,723 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:29,723 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,723 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:29,723 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:29,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:29,727 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:29,727 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:29,727 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:29,727 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:29,727 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:29,729 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,729 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,729 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,730 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,733 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,735 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,735 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,735 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,735 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,735 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,735 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,735 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,735 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,735 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,735 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:29,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:29,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:29,736 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:29,736 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:29,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288989817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288989817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:29,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288989823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,836 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:29,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:29,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:29,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:29,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:29,839 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:29,839 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,839 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:29,839 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:29,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,841 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,841 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,841 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,841 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:29,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:29,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:29,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:29,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:29,844 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,844 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:29,844 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:29,844 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:29,845 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,846 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,846 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f55cc4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:29,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:29,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,853 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,853 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,853 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:29,853 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:29,853 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:29,853 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:29,854 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:29,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:29,857 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:29,857 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:29,857 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:29,857 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:29,857 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:29,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,860 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,861 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:29,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,863 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,863 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:29,864 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:29,864 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:29,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,964 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:29,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:29,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:29,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:29,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:29,967 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:29,967 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,967 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:29,967 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:29,967 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,968 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,969 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,969 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:29,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:29,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:29,971 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,971 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:29,971 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:29,971 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:29,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:29,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,972 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,973 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,973 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:29,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b30529[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:29,974 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:29,974 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,975 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:29,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:29,975 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,975 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:29,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:29,977 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,977 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:29,977 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,977 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,977 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/b93d8079a25e40cd9783a63292d20fc1 2023-03-31 18:55:29,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:29,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:29,980 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:29,980 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:29,980 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:29,980 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:29,980 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:29,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/b3978dc77a0849d4b51c26dac8266405 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/b3978dc77a0849d4b51c26dac8266405 2023-03-31 18:55:29,982 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,982 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,982 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:29,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:29,983 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,984 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:29,985 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/b3978dc77a0849d4b51c26dac8266405, entries=400, sequenceid=310, filesize=59.8 K 2023-03-31 18:55:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,986 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/0da9ac087af54f748750405a83b9487e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0da9ac087af54f748750405a83b9487e 2023-03-31 18:55:29,986 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,986 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,986 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:29,986 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:29,987 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:29,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:29,987 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:29,987 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,987 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:29,987 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:29,987 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:29,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:29,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:29,987 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:29,987 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:29,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:29,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:29,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:29,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0da9ac087af54f748750405a83b9487e, entries=150, sequenceid=310, filesize=11.9 K 2023-03-31 18:55:29,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/b93d8079a25e40cd9783a63292d20fc1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b93d8079a25e40cd9783a63292d20fc1 2023-03-31 18:55:29,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b93d8079a25e40cd9783a63292d20fc1, entries=150, sequenceid=310, filesize=11.9 K 2023-03-31 18:55:29,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c2adb1ceb55b533a282bcfb4ded6271e in 880ms, sequenceid=310, compaction requested=true 2023-03-31 18:55:29,994 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:29,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:29,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:29,994 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:29,994 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:29,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:29,995 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:29,995 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:29,995 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/B is initiating minor compaction (all files) 2023-03-31 18:55:29,995 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/A is initiating minor compaction (all files) 2023-03-31 18:55:29,995 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/B in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,995 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/A in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:29,995 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/351430e1d42b443eb06c677c5956e422, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/128ec65b25a1414fb13baf85c752f479, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0da9ac087af54f748750405a83b9487e] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=35.9 K 2023-03-31 18:55:29,995 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ca519b5262a4a4593110db82e292c25, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/608cecfd4c3b448c9e498047d444889b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/b3978dc77a0849d4b51c26dac8266405] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=110.9 K 2023-03-31 18:55:29,996 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 351430e1d42b443eb06c677c5956e422, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680288926562 2023-03-31 18:55:29,996 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9ca519b5262a4a4593110db82e292c25, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680288926562 2023-03-31 18:55:29,996 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 128ec65b25a1414fb13baf85c752f479, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680288926894 2023-03-31 18:55:29,996 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 0da9ac087af54f748750405a83b9487e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680288929100 2023-03-31 18:55:29,996 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 608cecfd4c3b448c9e498047d444889b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680288926894 2023-03-31 18:55:29,997 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting b3978dc77a0849d4b51c26dac8266405, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680288929058 2023-03-31 18:55:30,001 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#B#compaction#315 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:30,006 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#A#compaction#316 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:30,087 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:30,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:30,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:30,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:30,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:30,091 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:30,091 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:30,091 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:30,091 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:30,091 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,091 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,092 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,092 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:30,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:30,094 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,094 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:30,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:30,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:30,096 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:30,097 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:30,097 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:30,097 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:30,097 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:30,097 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:30,097 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:30,097 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-31 18:55:30,097 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:30,098 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:30,098 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:30,098 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:30,098 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:30,098 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:30,110 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331bd648342603c43fd9267afa7605ce884 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331bd648342603c43fd9267afa7605ce884 2023-03-31 18:55:30,115 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=345, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/a72c28833468411a874220b2c0d1423e 2023-03-31 18:55:30,126 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/577ae23acd164896959a003722e258b2 2023-03-31 18:55:30,134 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/e9cda9f06a684ae883c31b5e223dedaa 2023-03-31 18:55:30,138 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/a72c28833468411a874220b2c0d1423e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a72c28833468411a874220b2c0d1423e 2023-03-31 18:55:30,141 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a72c28833468411a874220b2c0d1423e, entries=150, sequenceid=345, filesize=25.4 K 2023-03-31 18:55:30,142 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/577ae23acd164896959a003722e258b2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/577ae23acd164896959a003722e258b2 2023-03-31 18:55:30,145 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/577ae23acd164896959a003722e258b2, entries=150, sequenceid=345, filesize=11.9 K 2023-03-31 18:55:30,146 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/e9cda9f06a684ae883c31b5e223dedaa as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e9cda9f06a684ae883c31b5e223dedaa 2023-03-31 18:55:30,149 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e9cda9f06a684ae883c31b5e223dedaa, entries=150, sequenceid=345, filesize=11.9 K 2023-03-31 18:55:30,150 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for c2adb1ceb55b533a282bcfb4ded6271e in 53ms, sequenceid=345, compaction requested=true 2023-03-31 18:55:30,150 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:30,150 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:30,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:30,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:30,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:30,150 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:30,155 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,155 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:30,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:30,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@328931cc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:30,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:30,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,158 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,158 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:30,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:30,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:30,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:30,160 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:30,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:30,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:30,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:30,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:30,163 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:30,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:30,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:30,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:30,163 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:30,164 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,164 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,164 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:30,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:30,166 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:30,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:30,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:30,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:30,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:30,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:30,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:30,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,172 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:30,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:30,172 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:30,272 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:30,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:30,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:30,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:30,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:30,276 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:30,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:30,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:30,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:30,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,278 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:30,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:30,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:30,278 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,278 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,278 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:30,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:30,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:30,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:30,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:30,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:30,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:30,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:30,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:30,279 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:30,279 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:30,279 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:30,279 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:30,280 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:30,280 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:30,280 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,280 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:30,280 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:30,281 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,281 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:30,281 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,281 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:30,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:30,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:30,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57277cd4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:30,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,284 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:30,284 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:30,284 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:30,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:30,286 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:30,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:30,286 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:30,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:30,288 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:30,288 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:30,288 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:30,288 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:30,288 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:30,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:30,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:30,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:30,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:30,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:30,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:30,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:30,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:30,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,295 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:30,295 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,294 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:30,294 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:30,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:30,295 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:30,395 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:30,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:30,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:30,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:30,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:30,398 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:30,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:30,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:30,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:30,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,400 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:30,400 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:30,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:30,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:30,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:30,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:30,408 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:30,408 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:30,408 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:30,408 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:30,408 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:30,408 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:30,408 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:30,409 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-31 18:55:30,409 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:30,409 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:30,409 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:30,409 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:30,409 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:30,409 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:30,413 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/4c7b937bfba64b4bb5633b4398346b3c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c7b937bfba64b4bb5633b4398346b3c 2023-03-31 18:55:30,418 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/B of c2adb1ceb55b533a282bcfb4ded6271e into 4c7b937bfba64b4bb5633b4398346b3c(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:30,418 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:30,418 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/B, priority=13, startTime=1680288929994; duration=0sec 2023-03-31 18:55:30,418 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:30,418 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:30,419 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:30,419 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/C is initiating minor compaction (all files) 2023-03-31 18:55:30,419 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/C in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:30,419 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/a7c7b51f0a484838b6e036b110479ac7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ead9b4416d04105adeec04ac0fb20f2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b93d8079a25e40cd9783a63292d20fc1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e9cda9f06a684ae883c31b5e223dedaa] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=47.8 K 2023-03-31 18:55:30,420 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a7c7b51f0a484838b6e036b110479ac7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680288926562 2023-03-31 18:55:30,420 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9ead9b4416d04105adeec04ac0fb20f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680288926894 2023-03-31 18:55:30,420 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b93d8079a25e40cd9783a63292d20fc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680288929100 2023-03-31 18:55:30,425 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e9cda9f06a684ae883c31b5e223dedaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680288929189 2023-03-31 18:55:30,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:30,426 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:30,430 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331dac63a512b5d40c88e8a1d1948cd6d4f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331dac63a512b5d40c88e8a1d1948cd6d4f 2023-03-31 18:55:30,430 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/c60c4ca41f0c4f278d9ea2a7eabee680 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c60c4ca41f0c4f278d9ea2a7eabee680 2023-03-31 18:55:30,445 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#C#compaction#321 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:30,450 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/A of c2adb1ceb55b533a282bcfb4ded6271e into c60c4ca41f0c4f278d9ea2a7eabee680(size=25.8 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:30,451 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:30,451 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/A, priority=13, startTime=1680288929994; duration=0sec 2023-03-31 18:55:30,451 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:30,458 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=355, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/c21fa14ec3d540ab85e324fe5ce9697d 2023-03-31 18:55:30,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288930452, max timestamp=1680288930481Num uniques 859; 2023-03-31 18:55:30,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:55:30,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288930452, max timestamp=1680288930481Num uniques 859; 2023-03-31 18:55:30,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:55:30,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288930452, max timestamp=1680288930481Num uniques 859; 2023-03-31 18:55:30,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:55:30,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288990519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288990522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288990526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288990642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47904 deadline: 1680288990643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1680288990643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,654 DEBUG [Thread-2248] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:30,655 DEBUG [Thread-2254] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8221 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:30,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288990643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288990643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288990853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288990856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:30,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288990858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:30,862 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/54a0b047568446f0a97b32cf16a45dd6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/54a0b047568446f0a97b32cf16a45dd6 2023-03-31 18:55:30,866 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/C of c2adb1ceb55b533a282bcfb4ded6271e into 54a0b047568446f0a97b32cf16a45dd6(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:30,866 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:30,866 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/C, priority=12, startTime=1680288929994; duration=0sec 2023-03-31 18:55:30,866 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:30,867 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/065d4f4e742c499db9aff0779dc5b745 2023-03-31 18:55:31,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288991161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288991161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288991161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,277 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/57a3b1d1d4f34e85b666e569bed99720 2023-03-31 18:55:31,281 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/c21fa14ec3d540ab85e324fe5ce9697d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c21fa14ec3d540ab85e324fe5ce9697d 2023-03-31 18:55:31,284 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c21fa14ec3d540ab85e324fe5ce9697d, entries=250, sequenceid=355, filesize=39.2 K 2023-03-31 18:55:31,285 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/065d4f4e742c499db9aff0779dc5b745 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/065d4f4e742c499db9aff0779dc5b745 2023-03-31 18:55:31,288 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/065d4f4e742c499db9aff0779dc5b745, entries=150, sequenceid=355, filesize=11.9 K 2023-03-31 18:55:31,289 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/57a3b1d1d4f34e85b666e569bed99720 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/57a3b1d1d4f34e85b666e569bed99720 2023-03-31 18:55:31,292 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/57a3b1d1d4f34e85b666e569bed99720, entries=150, sequenceid=355, filesize=11.9 K 2023-03-31 18:55:31,293 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for c2adb1ceb55b533a282bcfb4ded6271e in 885ms, sequenceid=355, compaction requested=true 2023-03-31 18:55:31,293 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:31,293 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:31,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:31,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:31,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:31,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:31,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:31,296 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:31,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:31,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:31,297 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab91988[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:31,297 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:31,299 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:31,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:31,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:31,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:31,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:31,299 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,299 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:31,300 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,300 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:31,300 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:31,300 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:31,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:31,302 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:31,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:31,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:31,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:31,302 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:31,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:31,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:31,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:31,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:31,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:31,305 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,306 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,309 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:31,309 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:31,309 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:31,309 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:31,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:31,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:31,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:31,309 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:31,309 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:31,309 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:31,309 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:31,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:31,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:31,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:31,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:31,410 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:31,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:31,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:31,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:31,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:31,413 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:31,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:31,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:31,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:31,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:31,414 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:31,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:31,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:31,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:31,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:31,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:31,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:31,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:31,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:31,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:31,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:31,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:31,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:31,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:31,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:31,417 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-31 18:55:31,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:31,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:31,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:31,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:31,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:31,417 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:31,430 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331ef2610791ca14ebc8a578ab0e5ae1473 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ef2610791ca14ebc8a578ab0e5ae1473 2023-03-31 18:55:31,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:31,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:31,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288991674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288991676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288991677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288991781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288991781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288991784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,835 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=395, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/01bd8d0a7077403489cb6af630b2d174 2023-03-31 18:55:31,844 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/52d71498b9314177bd69cd4037a8eddf 2023-03-31 18:55:31,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288991989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288991989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:31,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:31,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288991991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,252 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/ae7bee12fc06496bbf7a33c4da4b3244 2023-03-31 18:55:32,256 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/01bd8d0a7077403489cb6af630b2d174 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/01bd8d0a7077403489cb6af630b2d174 2023-03-31 18:55:32,259 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/01bd8d0a7077403489cb6af630b2d174, entries=150, sequenceid=395, filesize=25.4 K 2023-03-31 18:55:32,259 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/52d71498b9314177bd69cd4037a8eddf as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/52d71498b9314177bd69cd4037a8eddf 2023-03-31 18:55:32,263 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/52d71498b9314177bd69cd4037a8eddf, entries=150, sequenceid=395, filesize=11.9 K 2023-03-31 18:55:32,263 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/ae7bee12fc06496bbf7a33c4da4b3244 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/ae7bee12fc06496bbf7a33c4da4b3244 2023-03-31 18:55:32,267 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/ae7bee12fc06496bbf7a33c4da4b3244, entries=150, sequenceid=395, filesize=11.9 K 2023-03-31 18:55:32,267 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for c2adb1ceb55b533a282bcfb4ded6271e in 850ms, sequenceid=395, compaction requested=true 2023-03-31 18:55:32,267 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:32,267 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,267 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:32,267 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:32,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:32,268 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:32,269 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,269 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,270 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,270 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:32,271 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:32,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57aed5bb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:32,271 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:32,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:32,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:32,273 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,273 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:32,278 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,278 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:32,283 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:32,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:32,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:32,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:32,284 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:32,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,286 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:32,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:32,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:32,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,289 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:32,289 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:32,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:32,297 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:55:32,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:32,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:32,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:32,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:32,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:32,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:32,314 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303318607d48a08a6482b8caf15a51b40e9c4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318607d48a08a6482b8caf15a51b40e9c4 2023-03-31 18:55:32,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288932329, max timestamp=1680288932367Num uniques 859; 2023-03-31 18:55:32,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:32,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288932329, max timestamp=1680288932367Num uniques 859; 2023-03-31 18:55:32,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:32,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288932329, max timestamp=1680288932367Num uniques 859; 2023-03-31 18:55:32,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:32,389 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:32,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:32,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:32,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:32,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:32,392 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:32,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:32,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:32,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:32,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,393 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,394 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,394 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,394 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,394 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:32,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:32,398 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,399 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:32,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5289d3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:32,400 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:32,400 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,401 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,401 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:32,401 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:32,401 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:32,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:32,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,402 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:32,402 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,402 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:32,404 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:32,404 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:32,404 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:32,404 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:32,404 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:32,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,411 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:32,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:32,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,411 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:32,411 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:32,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:32,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288992410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:32,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288992410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:32,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288992411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,511 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:32,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:32,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:32,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:32,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:32,514 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:32,515 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:32,515 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:32,515 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:32,515 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,515 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,516 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,516 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,517 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,517 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:32,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:32,518 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,518 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:32,518 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:32,518 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,519 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:32,519 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:32,520 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,520 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,520 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,520 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:32,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40ff8151[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:32,522 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:32,522 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,523 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:32,523 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,523 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:32,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:32,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288992519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:32,525 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:32,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288992519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,525 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,526 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:32,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288992520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:32,528 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:32,528 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:32,528 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:32,528 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:32,528 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:32,529 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,530 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,530 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,530 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,531 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,534 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:32,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:32,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,534 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,534 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,534 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,534 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,534 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:32,534 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:32,634 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:32,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:32,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:32,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:32,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:32,637 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:32,637 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:32,637 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:32,637 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:32,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,639 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:32,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:32,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:32,641 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,641 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:32,641 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:32,641 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:32,644 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,644 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,644 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,644 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:32,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5298b59f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:32,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:32,645 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,646 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:32,646 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:32,647 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:32,647 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,647 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:32,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:32,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,648 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,648 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:32,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:32,650 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:32,650 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:32,650 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:32,650 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:32,650 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:32,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,661 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,662 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,664 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:32,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:32,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:32,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,664 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:32,664 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:32,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,665 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:32,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288992726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:32,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288992727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:32,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288992728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,738 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=406, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/3a46d97763094d789c1bae505560cff1 2023-03-31 18:55:32,764 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:32,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:32,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:32,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:32,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:32,768 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:32,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:32,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:32,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:32,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,770 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,770 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:32,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:32,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:32,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:32,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:32,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:32,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:32,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:32,772 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,772 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:32,772 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:32,772 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:32,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:32,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73baa17a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:32,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:32,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,778 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:32,778 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,778 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:32,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:32,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:32,780 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:32,780 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,780 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:32,780 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:32,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:32,784 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:32,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:32,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:32,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:32,784 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:32,786 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,786 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,787 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,788 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,790 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,791 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:32,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:32,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,791 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:32,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,791 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:32,891 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:32,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:32,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:32,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:32,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:32,897 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:32,897 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:32,897 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:32,897 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:32,897 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,898 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,899 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:32,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:32,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:32,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:32,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:32,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:32,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:32,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:32,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:32,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:32,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,904 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,905 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,905 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9eb2cf[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:32,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:32,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,908 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:32,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:32,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:32,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:32,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:32,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:32,912 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:32,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:32,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:32,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:32,912 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:32,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:32,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,920 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:32,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:32,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:32,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:32,924 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:32,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:32,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:32,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:32,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:32,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:32,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:32,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:32,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:32,925 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:32,925 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:33,025 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:33,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:33,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:33,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:33,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:33,028 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:33,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:33,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:33,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:33,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,030 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,030 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:33,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:33,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:33,030 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,030 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:33,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:33,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:33,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:33,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:33,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:33,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:33,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:33,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:33,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:33,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:33,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288993028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:33,034 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,034 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:33,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:33,034 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,034 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:33,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:33,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:33,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:33,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:33,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@775757e4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:33,035 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:33,035 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,037 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:33,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:33,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:33,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:33,039 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:33,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:33,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:33,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:33,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:33,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288993037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:33,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:33,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,041 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:33,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:33,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:33,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:33,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288993037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,041 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:33,043 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,043 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,043 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:33,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:33,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,045 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:33,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:33,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:33,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:33,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:33,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:33,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,054 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:33,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,054 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:33,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:33,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/4b9495b540304026b12f271bfb7d2fb5 2023-03-31 18:55:33,154 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:33,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:33,158 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/260aaf4772f747ca8772a0c232685543 2023-03-31 18:55:33,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:33,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:33,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:33,159 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:33,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:33,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:33,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:33,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:33,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:33,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/3a46d97763094d789c1bae505560cff1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/3a46d97763094d789c1bae505560cff1 2023-03-31 18:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:33,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:33,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:33,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:33,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:33,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:33,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:33,163 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:33,163 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:33,163 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:33,163 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:33,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:33,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:33,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:33,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:33,164 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:33,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:33,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:33,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:33,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:33,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,166 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/3a46d97763094d789c1bae505560cff1, entries=250, sequenceid=406, filesize=39.2 K 2023-03-31 18:55:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:33,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:33,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@607430dc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:33,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/4b9495b540304026b12f271bfb7d2fb5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4b9495b540304026b12f271bfb7d2fb5 2023-03-31 18:55:33,166 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:33,167 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,168 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:33,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:33,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:33,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:33,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4b9495b540304026b12f271bfb7d2fb5, entries=150, sequenceid=406, filesize=11.9 K 2023-03-31 18:55:33,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/260aaf4772f747ca8772a0c232685543 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/260aaf4772f747ca8772a0c232685543 2023-03-31 18:55:33,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:33,175 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:33,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/260aaf4772f747ca8772a0c232685543, entries=150, sequenceid=406, filesize=11.9 K 2023-03-31 18:55:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:33,176 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c2adb1ceb55b533a282bcfb4ded6271e in 879ms, sequenceid=406, compaction requested=true 2023-03-31 18:55:33,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:33,176 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:55:33,176 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:55:33,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:33,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,178 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 158740 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:55:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:33,178 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/A is initiating minor compaction (all files) 2023-03-31 18:55:33,178 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:33,178 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/A in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:33,178 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:33,178 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:33,178 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:33,178 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:33,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:33,178 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61272 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:55:33,178 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c60c4ca41f0c4f278d9ea2a7eabee680, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a72c28833468411a874220b2c0d1423e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c21fa14ec3d540ab85e324fe5ce9697d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/01bd8d0a7077403489cb6af630b2d174, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/3a46d97763094d789c1bae505560cff1] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=155.0 K 2023-03-31 18:55:33,178 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/B is initiating minor compaction (all files) 2023-03-31 18:55:33,178 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/B in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:33,179 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c7b937bfba64b4bb5633b4398346b3c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/577ae23acd164896959a003722e258b2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/065d4f4e742c499db9aff0779dc5b745, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/52d71498b9314177bd69cd4037a8eddf, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4b9495b540304026b12f271bfb7d2fb5] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=59.8 K 2023-03-31 18:55:33,179 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c60c4ca41f0c4f278d9ea2a7eabee680, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680288929100 2023-03-31 18:55:33,179 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4c7b937bfba64b4bb5633b4398346b3c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680288929100 2023-03-31 18:55:33,179 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a72c28833468411a874220b2c0d1423e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680288929189 2023-03-31 18:55:33,179 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 577ae23acd164896959a003722e258b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680288929189 2023-03-31 18:55:33,179 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c21fa14ec3d540ab85e324fe5ce9697d, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680288930374 2023-03-31 18:55:33,180 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 065d4f4e742c499db9aff0779dc5b745, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680288930398 2023-03-31 18:55:33,180 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 01bd8d0a7077403489cb6af630b2d174, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680288930516 2023-03-31 18:55:33,180 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 52d71498b9314177bd69cd4037a8eddf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680288930516 2023-03-31 18:55:33,180 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 3a46d97763094d789c1bae505560cff1, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680288931673 2023-03-31 18:55:33,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,180 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4b9495b540304026b12f271bfb7d2fb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680288931675 2023-03-31 18:55:33,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:33,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:33,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:33,183 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,186 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:33,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:33,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:33,186 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:33,186 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:33,188 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:33,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:33,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:33,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:33,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:33,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:33,195 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#B#compaction#330 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:33,207 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#A#compaction#331 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:33,212 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/bf429c61f925433d9733bacb96459fe3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/bf429c61f925433d9733bacb96459fe3 2023-03-31 18:55:33,216 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/B of c2adb1ceb55b533a282bcfb4ded6271e into bf429c61f925433d9733bacb96459fe3(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:33,216 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:33,216 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/B, priority=11, startTime=1680288933176; duration=0sec 2023-03-31 18:55:33,216 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:33,216 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:33,217 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:33,217 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/C is initiating minor compaction (all files) 2023-03-31 18:55:33,217 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/C in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:33,217 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/54a0b047568446f0a97b32cf16a45dd6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/57a3b1d1d4f34e85b666e569bed99720, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/ae7bee12fc06496bbf7a33c4da4b3244, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/260aaf4772f747ca8772a0c232685543] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=48.0 K 2023-03-31 18:55:33,218 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 54a0b047568446f0a97b32cf16a45dd6, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680288929189 2023-03-31 18:55:33,218 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 57a3b1d1d4f34e85b666e569bed99720, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680288930398 2023-03-31 18:55:33,218 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ae7bee12fc06496bbf7a33c4da4b3244, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680288930516 2023-03-31 18:55:33,219 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 260aaf4772f747ca8772a0c232685543, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680288931675 2023-03-31 18:55:33,230 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#C#compaction#332 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:33,286 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:33,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:33,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:33,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:33,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:33,289 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:33,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:33,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:33,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:33,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,292 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:33,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:33,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:33,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:33,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:33,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:33,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:33,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:33,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:33,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:33,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:33,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:33,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:33,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:33,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:33,294 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-31 18:55:33,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:33,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:33,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:33,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:33,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:33,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:33,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:33,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:33,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288993558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288993559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288993561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,647 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/795c7ad1b3254dcf9a844e1efab264a8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/795c7ad1b3254dcf9a844e1efab264a8 2023-03-31 18:55:33,651 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/C of c2adb1ceb55b533a282bcfb4ded6271e into 795c7ad1b3254dcf9a844e1efab264a8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:33,651 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:33,651 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/C, priority=12, startTime=1680288933176; duration=0sec 2023-03-31 18:55:33,651 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:33,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288993669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288993671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288993673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,711 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303314d968589d1484bc6ae07126cfc9cc0e2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303314d968589d1484bc6ae07126cfc9cc0e2 2023-03-31 18:55:33,717 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=441, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/6764131e5e1e42bfaf2416e7ed2630c9 2023-03-31 18:55:33,728 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/d4d9e775489c4acfb4495c301d5be1da 2023-03-31 18:55:33,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288993876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288993878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:33,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:33,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288993879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,047 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/26a8b1cf66f9469ba71ca8a14e325d5e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/26a8b1cf66f9469ba71ca8a14e325d5e 2023-03-31 18:55:34,051 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/A of c2adb1ceb55b533a282bcfb4ded6271e into 26a8b1cf66f9469ba71ca8a14e325d5e(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:34,051 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:34,051 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/A, priority=11, startTime=1680288933176; duration=0sec 2023-03-31 18:55:34,051 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:34,148 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/91c4902bbc2147e085aebe2ba13a482f 2023-03-31 18:55:34,152 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/6764131e5e1e42bfaf2416e7ed2630c9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/6764131e5e1e42bfaf2416e7ed2630c9 2023-03-31 18:55:34,156 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/6764131e5e1e42bfaf2416e7ed2630c9, entries=150, sequenceid=441, filesize=25.4 K 2023-03-31 18:55:34,157 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/d4d9e775489c4acfb4495c301d5be1da as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d4d9e775489c4acfb4495c301d5be1da 2023-03-31 18:55:34,162 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d4d9e775489c4acfb4495c301d5be1da, entries=150, sequenceid=441, filesize=11.9 K 2023-03-31 18:55:34,163 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/91c4902bbc2147e085aebe2ba13a482f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91c4902bbc2147e085aebe2ba13a482f 2023-03-31 18:55:34,169 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91c4902bbc2147e085aebe2ba13a482f, entries=150, sequenceid=441, filesize=11.9 K 2023-03-31 18:55:34,170 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for c2adb1ceb55b533a282bcfb4ded6271e in 876ms, sequenceid=441, compaction requested=false 2023-03-31 18:55:34,170 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:34,170 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:34,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:34,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:34,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:34,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:34,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,178 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:34,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46e3ecd9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:34,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:34,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:34,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:34,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:34,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:34,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:34,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:34,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:34,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:34,185 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:34,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:34,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:34,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:34,185 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:34,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:34,188 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,189 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:34,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:34,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:34,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:34,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:34,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:34,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:34,193 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:34,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,193 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:34,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:34,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:34,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:55:34,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:34,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:34,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:34,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:34,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:34,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:34,217 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033140fd5f2dbdb449609c20885703c2469d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033140fd5f2dbdb449609c20885703c2469d 2023-03-31 18:55:34,228 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=455, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/925b231e7812464ea2ff461e9e75b64d 2023-03-31 18:55:34,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288994242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288994242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288994248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,294 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:34,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:34,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:34,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:34,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:34,298 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:34,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:34,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:34,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:34,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,299 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:34,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:34,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:34,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:34,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:34,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:34,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:34,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:34,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:34,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:34,303 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:34,304 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:34,304 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:34,304 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:34,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:34,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:34,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:34,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:34,305 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,305 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,306 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,306 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:34,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78d53600[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:34,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:34,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:34,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:34,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:34,308 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,308 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:34,314 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:34,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:34,317 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:34,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:34,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:34,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:34,317 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:34,319 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,319 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:34,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,319 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:34,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:34,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:34,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,329 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:34,329 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:34,330 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:34,330 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:34,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:34,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:34,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288994353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288994353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288994354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,429 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:34,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:34,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:34,432 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:34,432 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:34,432 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:34,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:34,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,436 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:34,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:34,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:34,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:34,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:34,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:34,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:34,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:34,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:34,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:34,437 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:34,438 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:34,438 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:34,438 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:34,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:34,439 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,439 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,440 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,440 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:34,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3570b689[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:34,441 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:34,441 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,442 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:34,442 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:34,442 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:34,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:34,443 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:34,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:34,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:34,448 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:34,448 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:34,448 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:34,448 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:34,448 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:34,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:34,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,455 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,455 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,455 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:34,455 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:34,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:34,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:34,455 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,455 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,455 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,455 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,455 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,455 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:34,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:34,455 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:34,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:34,455 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:34,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:34,555 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:34,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:34,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:34,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:34,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:34,561 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:34,561 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:34,561 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:34,561 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:34,561 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,561 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,562 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:34,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:34,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:34,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288994559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288994559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288994560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:34,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:34,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:34,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:34,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:34,573 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:34,573 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:34,573 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:34,573 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:34,575 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:34,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c3822b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:34,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:34,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:34,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:34,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:34,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:34,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:34,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:34,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:34,581 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:34,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:34,583 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:34,583 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:34,583 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:34,583 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:34,583 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:34,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:34,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:34,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:34,585 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,587 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:34,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:34,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:34,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:34,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:34,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:34,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:34,590 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:34,590 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:34,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,590 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:34,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:34,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:34,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/fa44bf5cd67c4c38b74185b116d6ddd2 2023-03-31 18:55:34,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/d81a5d41bfef4c5282ef2bd625ca8109 2023-03-31 18:55:34,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/925b231e7812464ea2ff461e9e75b64d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/925b231e7812464ea2ff461e9e75b64d 2023-03-31 18:55:34,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/925b231e7812464ea2ff461e9e75b64d, entries=150, sequenceid=455, filesize=25.4 K 2023-03-31 18:55:34,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/fa44bf5cd67c4c38b74185b116d6ddd2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/fa44bf5cd67c4c38b74185b116d6ddd2 2023-03-31 18:55:34,656 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/fa44bf5cd67c4c38b74185b116d6ddd2, entries=150, sequenceid=455, filesize=11.9 K 2023-03-31 18:55:34,657 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/d81a5d41bfef4c5282ef2bd625ca8109 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/d81a5d41bfef4c5282ef2bd625ca8109 2023-03-31 18:55:34,660 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/d81a5d41bfef4c5282ef2bd625ca8109, entries=150, sequenceid=455, filesize=11.9 K 2023-03-31 18:55:34,661 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c2adb1ceb55b533a282bcfb4ded6271e in 464ms, sequenceid=455, compaction requested=true 2023-03-31 18:55:34,661 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:34,661 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:34,661 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:34,661 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:34,661 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:34,661 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:34,662 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78650 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:34,662 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:34,662 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/A is initiating minor compaction (all files) 2023-03-31 18:55:34,662 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/B is initiating minor compaction (all files) 2023-03-31 18:55:34,662 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/A in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:34,662 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/B in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:34,662 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/26a8b1cf66f9469ba71ca8a14e325d5e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/6764131e5e1e42bfaf2416e7ed2630c9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/925b231e7812464ea2ff461e9e75b64d] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=76.8 K 2023-03-31 18:55:34,662 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/bf429c61f925433d9733bacb96459fe3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d4d9e775489c4acfb4495c301d5be1da, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/fa44bf5cd67c4c38b74185b116d6ddd2] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=36.2 K 2023-03-31 18:55:34,662 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 26a8b1cf66f9469ba71ca8a14e325d5e, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680288931675 2023-03-31 18:55:34,662 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting bf429c61f925433d9733bacb96459fe3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680288931675 2023-03-31 18:55:34,663 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 6764131e5e1e42bfaf2416e7ed2630c9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680288932395 2023-03-31 18:55:34,663 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d4d9e775489c4acfb4495c301d5be1da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680288932395 2023-03-31 18:55:34,663 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 925b231e7812464ea2ff461e9e75b64d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680288933558 2023-03-31 18:55:34,664 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting fa44bf5cd67c4c38b74185b116d6ddd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680288933558 2023-03-31 18:55:34,669 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#B#compaction#340 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:34,671 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#A#compaction#339 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:34,690 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:34,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:34,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:34,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:34,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:34,693 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:34,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:34,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:34,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:34,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:34,697 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:34,697 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,697 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:34,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:34,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:34,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:34,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:34,699 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:34,699 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:34,699 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:34,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:34,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:34,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:34,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:34,704 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:55:34,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:34,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:34,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:34,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:34,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:34,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:34,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:34,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:34,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288994886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288994893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288994893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288994994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:34,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:34,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288994996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:35,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288995000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,079 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/ba5df1902a22468aab2a31e7ffd3f02a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/ba5df1902a22468aab2a31e7ffd3f02a 2023-03-31 18:55:35,084 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/B of c2adb1ceb55b533a282bcfb4ded6271e into ba5df1902a22468aab2a31e7ffd3f02a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:35,084 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:35,084 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/B, priority=13, startTime=1680288934661; duration=0sec 2023-03-31 18:55:35,084 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:35,084 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:55:35,086 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:55:35,086 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/C is initiating minor compaction (all files) 2023-03-31 18:55:35,087 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/C in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:35,087 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/795c7ad1b3254dcf9a844e1efab264a8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91c4902bbc2147e085aebe2ba13a482f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/d81a5d41bfef4c5282ef2bd625ca8109] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=36.2 K 2023-03-31 18:55:35,087 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 795c7ad1b3254dcf9a844e1efab264a8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680288931675 2023-03-31 18:55:35,088 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 91c4902bbc2147e085aebe2ba13a482f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680288932395 2023-03-31 18:55:35,088 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d81a5d41bfef4c5282ef2bd625ca8109, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680288933558 2023-03-31 18:55:35,101 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#C#compaction#342 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:35,136 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331397a4923eee5460b9f7c1dec4e8cf233 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331397a4923eee5460b9f7c1dec4e8cf233 2023-03-31 18:55:35,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:35,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288995200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:35,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288995202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:35,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288995207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:35,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288995506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:35,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288995509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,521 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/e5b6c4ce2f8241c98d36503c1fd28df4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e5b6c4ce2f8241c98d36503c1fd28df4 2023-03-31 18:55:35,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:35,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288995517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,526 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/C of c2adb1ceb55b533a282bcfb4ded6271e into e5b6c4ce2f8241c98d36503c1fd28df4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:35,526 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:35,526 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/C, priority=13, startTime=1680288934661; duration=0sec 2023-03-31 18:55:35,526 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:35,562 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/cbd9ffc0fa5047878ab64ba7608e7349 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/cbd9ffc0fa5047878ab64ba7608e7349 2023-03-31 18:55:35,566 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=475, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/a5b7074d3e4645249d1918cc70adffd2 2023-03-31 18:55:35,566 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/A of c2adb1ceb55b533a282bcfb4ded6271e into cbd9ffc0fa5047878ab64ba7608e7349(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:35,566 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:35,566 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/A, priority=13, startTime=1680288934661; duration=0sec 2023-03-31 18:55:35,566 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:35,614 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/4c1e8fbeee5b44e2a8bc9d6434e924ae 2023-03-31 18:55:35,634 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/2155a03e8b3f46cbb72aa33cd7b76092 2023-03-31 18:55:35,638 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/a5b7074d3e4645249d1918cc70adffd2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a5b7074d3e4645249d1918cc70adffd2 2023-03-31 18:55:35,644 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a5b7074d3e4645249d1918cc70adffd2, entries=150, sequenceid=475, filesize=25.4 K 2023-03-31 18:55:35,644 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/4c1e8fbeee5b44e2a8bc9d6434e924ae as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c1e8fbeee5b44e2a8bc9d6434e924ae 2023-03-31 18:55:35,648 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c1e8fbeee5b44e2a8bc9d6434e924ae, entries=150, sequenceid=475, filesize=11.9 K 2023-03-31 18:55:35,649 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/2155a03e8b3f46cbb72aa33cd7b76092 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/2155a03e8b3f46cbb72aa33cd7b76092 2023-03-31 18:55:35,661 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/2155a03e8b3f46cbb72aa33cd7b76092, entries=150, sequenceid=475, filesize=11.9 K 2023-03-31 18:55:35,662 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-31 18:55:35,662 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c2adb1ceb55b533a282bcfb4ded6271e in 958ms, sequenceid=475, compaction requested=false 2023-03-31 18:55:35,662 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:35,662 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:35,663 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:35,663 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,663 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:35,663 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:35,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:35,665 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:35,666 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:35,666 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:35,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:35,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d9f3d4e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:35,668 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:35,668 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:35,669 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:35,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:35,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:35,669 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:35,669 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:35,669 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:35,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:35,670 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:35,670 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:35,671 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:35,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:35,674 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:35,674 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:35,674 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:35,675 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:35,675 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:35,680 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:35,680 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:35,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:35,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:35,681 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,682 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,684 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:35,685 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,685 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:35,685 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:35,685 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:35,685 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:35,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:35,685 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,686 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:35,686 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:35,686 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:35,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:35,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:35,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:35,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:35,686 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:35,686 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:35,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:35,786 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:35,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:35,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:35,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:35,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:35,789 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:35,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:35,790 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:35,790 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:35,790 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:35,790 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:35,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:35,791 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:35,792 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:35,792 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:35,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:35,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:35,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:35,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:35,793 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:35,793 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:35,793 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:35,793 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:35,793 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:35,793 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:35,793 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:35,793 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:55:35,794 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:35,794 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:35,794 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:35,794 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:35,794 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:35,794 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:36,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:36,015 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:36,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288996043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288996043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288996044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288996151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288996151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288996151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,206 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331e1b01804f59248dd9819ebb8259b1b4b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e1b01804f59248dd9819ebb8259b1b4b 2023-03-31 18:55:36,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288996355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288996355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288996355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,613 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=490, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/114d91339c334f1eadb7750464767061 2023-03-31 18:55:36,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288996662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288996662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:36,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:36,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288996662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,024 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/1466889453484929a4a351b2f41bfd84 2023-03-31 18:55:37,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:37,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288997163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:37,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288997165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:37,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288997165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,434 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/91d71d74c6904e2e8ab77c90d8a43523 2023-03-31 18:55:37,437 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/114d91339c334f1eadb7750464767061 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/114d91339c334f1eadb7750464767061 2023-03-31 18:55:37,441 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/114d91339c334f1eadb7750464767061, entries=150, sequenceid=490, filesize=25.4 K 2023-03-31 18:55:37,442 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/1466889453484929a4a351b2f41bfd84 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1466889453484929a4a351b2f41bfd84 2023-03-31 18:55:37,445 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1466889453484929a4a351b2f41bfd84, entries=150, sequenceid=490, filesize=11.9 K 2023-03-31 18:55:37,445 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/91d71d74c6904e2e8ab77c90d8a43523 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91d71d74c6904e2e8ab77c90d8a43523 2023-03-31 18:55:37,449 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91d71d74c6904e2e8ab77c90d8a43523, entries=150, sequenceid=490, filesize=11.9 K 2023-03-31 18:55:37,449 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c2adb1ceb55b533a282bcfb4ded6271e in 1656ms, sequenceid=490, compaction requested=true 2023-03-31 18:55:37,449 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:37,449 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:37,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:37,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:37,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:37,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:37,459 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,459 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:37,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:37,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:37,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:37,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:37,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:37,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:37,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:37,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:37,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:37,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cab6bcb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:37,461 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:37,461 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:37,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:37,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:37,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:37,462 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:37,462 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:37,462 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:37,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:37,463 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:37,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,463 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:37,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:37,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:37,466 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:37,466 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:37,466 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:37,466 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:37,466 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:37,469 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:37,469 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:37,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:37,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:37,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:37,469 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:37,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:37,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:37,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:37,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,472 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,473 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:37,474 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:37,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:37,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:37,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:37,474 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:37,474 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:37,474 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:37,474 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:37,474 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:37,474 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:37,474 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:37,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:37,474 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:37,474 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:37,474 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:37,574 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:37,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:37,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:37,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:37,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:37,577 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:37,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:37,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:37,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:37,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:37,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:37,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:37,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:37,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:37,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:37,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:37,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:37,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:37,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:37,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:37,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:37,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:37,581 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:37,581 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:37,581 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:55:37,581 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:37,581 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:37,581 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:37,581 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:37,581 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:37,581 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:37,683 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-31 18:55:37,711 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/09160208d1884418a1f31863e5d1889b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/103ae98d87764153b56ecff0d261751e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/22e75b96ed444a26adb88727bdf4b508, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/34c59f49b9924b5b99ca1003aed63d1b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/583034e6e8b249f39c29df398fc39fbe, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/4c9cc339e26d4059915af9c2591e071d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/294d331b0c184f4ebc28c6a48b312084, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ca519b5262a4a4593110db82e292c25, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/608cecfd4c3b448c9e498047d444889b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/b3978dc77a0849d4b51c26dac8266405, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c60c4ca41f0c4f278d9ea2a7eabee680, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a72c28833468411a874220b2c0d1423e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c21fa14ec3d540ab85e324fe5ce9697d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/01bd8d0a7077403489cb6af630b2d174, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/3a46d97763094d789c1bae505560cff1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/26a8b1cf66f9469ba71ca8a14e325d5e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/6764131e5e1e42bfaf2416e7ed2630c9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/925b231e7812464ea2ff461e9e75b64d] to archive 2023-03-31 18:55:37,712 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:55:37,714 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/09160208d1884418a1f31863e5d1889b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/09160208d1884418a1f31863e5d1889b 2023-03-31 18:55:37,715 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/103ae98d87764153b56ecff0d261751e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/103ae98d87764153b56ecff0d261751e 2023-03-31 18:55:37,717 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/22e75b96ed444a26adb88727bdf4b508 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/22e75b96ed444a26adb88727bdf4b508 2023-03-31 18:55:37,718 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/34c59f49b9924b5b99ca1003aed63d1b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/34c59f49b9924b5b99ca1003aed63d1b 2023-03-31 18:55:37,719 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/583034e6e8b249f39c29df398fc39fbe to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/583034e6e8b249f39c29df398fc39fbe 2023-03-31 18:55:37,721 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/4c9cc339e26d4059915af9c2591e071d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/4c9cc339e26d4059915af9c2591e071d 2023-03-31 18:55:37,722 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/294d331b0c184f4ebc28c6a48b312084 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/294d331b0c184f4ebc28c6a48b312084 2023-03-31 18:55:37,723 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ca519b5262a4a4593110db82e292c25 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ca519b5262a4a4593110db82e292c25 2023-03-31 18:55:37,724 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/608cecfd4c3b448c9e498047d444889b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/608cecfd4c3b448c9e498047d444889b 2023-03-31 18:55:37,725 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/b3978dc77a0849d4b51c26dac8266405 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/b3978dc77a0849d4b51c26dac8266405 2023-03-31 18:55:37,727 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c60c4ca41f0c4f278d9ea2a7eabee680 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c60c4ca41f0c4f278d9ea2a7eabee680 2023-03-31 18:55:37,728 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a72c28833468411a874220b2c0d1423e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a72c28833468411a874220b2c0d1423e 2023-03-31 18:55:37,729 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c21fa14ec3d540ab85e324fe5ce9697d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/c21fa14ec3d540ab85e324fe5ce9697d 2023-03-31 18:55:37,730 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/01bd8d0a7077403489cb6af630b2d174 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/01bd8d0a7077403489cb6af630b2d174 2023-03-31 18:55:37,731 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/3a46d97763094d789c1bae505560cff1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/3a46d97763094d789c1bae505560cff1 2023-03-31 18:55:37,732 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/26a8b1cf66f9469ba71ca8a14e325d5e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/26a8b1cf66f9469ba71ca8a14e325d5e 2023-03-31 18:55:37,733 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/6764131e5e1e42bfaf2416e7ed2630c9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/6764131e5e1e42bfaf2416e7ed2630c9 2023-03-31 18:55:37,733 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/925b231e7812464ea2ff461e9e75b64d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/925b231e7812464ea2ff461e9e75b64d 2023-03-31 18:55:37,759 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/96b2f9675ad947bfba0a68ee85385546, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/b16411266f594eedaf1ef102c67c7390, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/dc491004760541f7904d3a284a3f1485, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/8e42cf61dbd6465095181ac043f9b58e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/93349635496a4a87bc090a3b29cd0142, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/13f29aa4c52348c4b4f42bfab84d5ce5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/351430e1d42b443eb06c677c5956e422, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1f605752eb7f41a7b164a283d14476e1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/128ec65b25a1414fb13baf85c752f479, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c7b937bfba64b4bb5633b4398346b3c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0da9ac087af54f748750405a83b9487e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/577ae23acd164896959a003722e258b2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/065d4f4e742c499db9aff0779dc5b745, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/52d71498b9314177bd69cd4037a8eddf, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/bf429c61f925433d9733bacb96459fe3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4b9495b540304026b12f271bfb7d2fb5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d4d9e775489c4acfb4495c301d5be1da, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/fa44bf5cd67c4c38b74185b116d6ddd2] to archive 2023-03-31 18:55:37,760 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:55:37,762 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/96b2f9675ad947bfba0a68ee85385546 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/96b2f9675ad947bfba0a68ee85385546 2023-03-31 18:55:37,763 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/b16411266f594eedaf1ef102c67c7390 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/b16411266f594eedaf1ef102c67c7390 2023-03-31 18:55:37,764 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/dc491004760541f7904d3a284a3f1485 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/dc491004760541f7904d3a284a3f1485 2023-03-31 18:55:37,765 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/8e42cf61dbd6465095181ac043f9b58e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/8e42cf61dbd6465095181ac043f9b58e 2023-03-31 18:55:37,766 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/93349635496a4a87bc090a3b29cd0142 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/93349635496a4a87bc090a3b29cd0142 2023-03-31 18:55:37,767 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/13f29aa4c52348c4b4f42bfab84d5ce5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/13f29aa4c52348c4b4f42bfab84d5ce5 2023-03-31 18:55:37,768 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/351430e1d42b443eb06c677c5956e422 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/351430e1d42b443eb06c677c5956e422 2023-03-31 18:55:37,769 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1f605752eb7f41a7b164a283d14476e1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1f605752eb7f41a7b164a283d14476e1 2023-03-31 18:55:37,770 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/128ec65b25a1414fb13baf85c752f479 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/128ec65b25a1414fb13baf85c752f479 2023-03-31 18:55:37,771 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c7b937bfba64b4bb5633b4398346b3c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c7b937bfba64b4bb5633b4398346b3c 2023-03-31 18:55:37,772 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0da9ac087af54f748750405a83b9487e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0da9ac087af54f748750405a83b9487e 2023-03-31 18:55:37,773 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/577ae23acd164896959a003722e258b2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/577ae23acd164896959a003722e258b2 2023-03-31 18:55:37,774 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/065d4f4e742c499db9aff0779dc5b745 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/065d4f4e742c499db9aff0779dc5b745 2023-03-31 18:55:37,775 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/52d71498b9314177bd69cd4037a8eddf to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/52d71498b9314177bd69cd4037a8eddf 2023-03-31 18:55:37,776 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/bf429c61f925433d9733bacb96459fe3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/bf429c61f925433d9733bacb96459fe3 2023-03-31 18:55:37,777 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4b9495b540304026b12f271bfb7d2fb5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4b9495b540304026b12f271bfb7d2fb5 2023-03-31 18:55:37,778 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d4d9e775489c4acfb4495c301d5be1da to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d4d9e775489c4acfb4495c301d5be1da 2023-03-31 18:55:37,779 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/fa44bf5cd67c4c38b74185b116d6ddd2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/fa44bf5cd67c4c38b74185b116d6ddd2 2023-03-31 18:55:37,784 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/4ee0acb8a12c4f6091cea23083e31675, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/86a2e154eb0b48acbbcce57fa5877d4f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/23383f6922ed416dbdfebeeec8edc859, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b19d32f21322449cb802781c964811e3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/116c9585730f4b4880e5a08438f362e8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/8fe4be991cb047d99e4ba58ce6edbe34, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/a7c7b51f0a484838b6e036b110479ac7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ec3b4a0b5014d69843cbfb981331b75, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ead9b4416d04105adeec04ac0fb20f2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b93d8079a25e40cd9783a63292d20fc1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/54a0b047568446f0a97b32cf16a45dd6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e9cda9f06a684ae883c31b5e223dedaa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/57a3b1d1d4f34e85b666e569bed99720, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/ae7bee12fc06496bbf7a33c4da4b3244, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/795c7ad1b3254dcf9a844e1efab264a8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/260aaf4772f747ca8772a0c232685543, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91c4902bbc2147e085aebe2ba13a482f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/d81a5d41bfef4c5282ef2bd625ca8109] to archive 2023-03-31 18:55:37,785 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:55:37,786 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/4ee0acb8a12c4f6091cea23083e31675 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/4ee0acb8a12c4f6091cea23083e31675 2023-03-31 18:55:37,788 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/86a2e154eb0b48acbbcce57fa5877d4f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/86a2e154eb0b48acbbcce57fa5877d4f 2023-03-31 18:55:37,789 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/23383f6922ed416dbdfebeeec8edc859 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/23383f6922ed416dbdfebeeec8edc859 2023-03-31 18:55:37,790 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b19d32f21322449cb802781c964811e3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b19d32f21322449cb802781c964811e3 2023-03-31 18:55:37,790 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/116c9585730f4b4880e5a08438f362e8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/116c9585730f4b4880e5a08438f362e8 2023-03-31 18:55:37,792 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/8fe4be991cb047d99e4ba58ce6edbe34 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/8fe4be991cb047d99e4ba58ce6edbe34 2023-03-31 18:55:37,793 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/a7c7b51f0a484838b6e036b110479ac7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/a7c7b51f0a484838b6e036b110479ac7 2023-03-31 18:55:37,794 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ec3b4a0b5014d69843cbfb981331b75 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ec3b4a0b5014d69843cbfb981331b75 2023-03-31 18:55:37,795 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ead9b4416d04105adeec04ac0fb20f2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9ead9b4416d04105adeec04ac0fb20f2 2023-03-31 18:55:37,796 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b93d8079a25e40cd9783a63292d20fc1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/b93d8079a25e40cd9783a63292d20fc1 2023-03-31 18:55:37,797 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/54a0b047568446f0a97b32cf16a45dd6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/54a0b047568446f0a97b32cf16a45dd6 2023-03-31 18:55:37,798 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e9cda9f06a684ae883c31b5e223dedaa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e9cda9f06a684ae883c31b5e223dedaa 2023-03-31 18:55:37,799 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/57a3b1d1d4f34e85b666e569bed99720 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/57a3b1d1d4f34e85b666e569bed99720 2023-03-31 18:55:37,800 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/ae7bee12fc06496bbf7a33c4da4b3244 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/ae7bee12fc06496bbf7a33c4da4b3244 2023-03-31 18:55:37,801 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/795c7ad1b3254dcf9a844e1efab264a8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/795c7ad1b3254dcf9a844e1efab264a8 2023-03-31 18:55:37,802 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/260aaf4772f747ca8772a0c232685543 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/260aaf4772f747ca8772a0c232685543 2023-03-31 18:55:37,804 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91c4902bbc2147e085aebe2ba13a482f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91c4902bbc2147e085aebe2ba13a482f 2023-03-31 18:55:37,805 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/d81a5d41bfef4c5282ef2bd625ca8109 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/d81a5d41bfef4c5282ef2bd625ca8109 2023-03-31 18:55:37,997 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331b92dfbb9fa9146f0882a9ecbfcd179b9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b92dfbb9fa9146f0882a9ecbfcd179b9 2023-03-31 18:55:38,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:38,173 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:38,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288998186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288998187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288998189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288998290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288998293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288998294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,402 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=510, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/129d55b0252d4773a8c5d61a9ee6eadc 2023-03-31 18:55:38,413 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/a44e0954c9724ad89d00a65ab14a47d0 2023-03-31 18:55:38,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288998497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288998502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288998502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288998802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288998806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:38,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288998807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,825 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/68550689c934462cbc61a6a489776daf 2023-03-31 18:55:38,828 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/129d55b0252d4773a8c5d61a9ee6eadc as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/129d55b0252d4773a8c5d61a9ee6eadc 2023-03-31 18:55:38,832 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/129d55b0252d4773a8c5d61a9ee6eadc, entries=150, sequenceid=510, filesize=25.4 K 2023-03-31 18:55:38,833 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/a44e0954c9724ad89d00a65ab14a47d0 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a44e0954c9724ad89d00a65ab14a47d0 2023-03-31 18:55:38,836 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a44e0954c9724ad89d00a65ab14a47d0, entries=150, sequenceid=510, filesize=11.9 K 2023-03-31 18:55:38,837 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/68550689c934462cbc61a6a489776daf as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/68550689c934462cbc61a6a489776daf 2023-03-31 18:55:38,842 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/68550689c934462cbc61a6a489776daf, entries=150, sequenceid=510, filesize=11.9 K 2023-03-31 18:55:38,842 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c2adb1ceb55b533a282bcfb4ded6271e in 1261ms, sequenceid=510, compaction requested=true 2023-03-31 18:55:38,843 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:38,843 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:38,844 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:38,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:38,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c5f5e29[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:38,846 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:38,846 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:38,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:38,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:38,849 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:38,849 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:38,849 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:38,849 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,849 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:38,850 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,850 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:38,850 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:38,850 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:38,853 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:38,853 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:38,853 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:38,853 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:38,853 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:38,855 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:38,855 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:38,855 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:38,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:38,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:38,855 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,857 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:38,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:38,859 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:38,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:38,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:38,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:38,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:38,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:38,859 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:38,859 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:38,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:38,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:38,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:38,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:38,959 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:38,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:38,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:38,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:38,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:38,962 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:38,962 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:38,962 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:38,962 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:38,963 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:38,963 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:38,966 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:38,966 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:38,966 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:38,966 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:38,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:38,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:38,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:38,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:38,967 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:38,967 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:38,967 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:38,968 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:38,968 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:38,968 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:38,968 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:38,968 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:55:38,968 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:38,968 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:38,968 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:38,968 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:38,968 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:38,968 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:38,979 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331512af9348fba4262b8dcda00fd2ff11f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331512af9348fba4262b8dcda00fd2ff11f 2023-03-31 18:55:38,984 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=522, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/231edcdadfe44a14be9a4df20f1af062 2023-03-31 18:55:39,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:39,330 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:39,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288999374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288999377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288999384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,393 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/6e1f49d272f24c12b84f8cc5c63137cf 2023-03-31 18:55:39,419 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/1001a91ffc844301a69c54beebb9bbba 2023-03-31 18:55:39,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/231edcdadfe44a14be9a4df20f1af062 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/231edcdadfe44a14be9a4df20f1af062 2023-03-31 18:55:39,430 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/231edcdadfe44a14be9a4df20f1af062, entries=100, sequenceid=522, filesize=18.6 K 2023-03-31 18:55:39,431 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/6e1f49d272f24c12b84f8cc5c63137cf as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/6e1f49d272f24c12b84f8cc5c63137cf 2023-03-31 18:55:39,435 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/6e1f49d272f24c12b84f8cc5c63137cf, entries=100, sequenceid=522, filesize=9.5 K 2023-03-31 18:55:39,436 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/1001a91ffc844301a69c54beebb9bbba as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/1001a91ffc844301a69c54beebb9bbba 2023-03-31 18:55:39,444 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/1001a91ffc844301a69c54beebb9bbba, entries=100, sequenceid=522, filesize=9.5 K 2023-03-31 18:55:39,444 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c2adb1ceb55b533a282bcfb4ded6271e in 476ms, sequenceid=522, compaction requested=true 2023-03-31 18:55:39,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:39,444 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:39,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:39,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:39,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:39,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:39,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:39,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24655f53[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:39,448 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:39,448 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:39,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:39,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:39,450 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,450 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:39,451 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:39,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:39,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:39,454 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:39,455 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:39,455 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:39,455 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:39,455 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:39,459 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,459 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,459 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:39,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,461 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:39,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:39,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:39,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:39,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:39,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:39,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,464 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:39,464 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:39,464 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,464 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:39,464 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:39,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-31 18:55:39,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288939371, max timestamp=1680288939492Num uniques 859; 2023-03-31 18:55:39,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:55:39,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288939371, max timestamp=1680288939492Num uniques 859; 2023-03-31 18:55:39,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:55:39,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:39,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:39,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:39,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:39,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:39,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:39,519 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303316af901a6119e4cfaa8d600ac95280a0d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303316af901a6119e4cfaa8d600ac95280a0d 2023-03-31 18:55:39,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288999532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288999539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288999540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,564 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:39,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:39,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:39,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:39,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:39,567 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:39,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:39,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:39,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:39,568 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,568 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:39,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:39,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:39,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:39,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:39,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:39,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:39,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:39,570 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:39,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:39,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:39,571 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:39,571 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:39,571 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:39,571 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:39,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:39,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:39,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:39,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:39,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,572 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:39,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:39,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70c473dc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:39,574 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:39,574 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,578 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,579 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:39,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:39,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:39,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:39,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:39,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:39,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:39,582 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:39,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:39,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:39,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:39,582 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:39,583 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,583 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:39,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,583 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:39,584 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,585 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:39,588 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:39,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:39,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:39,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:39,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:39,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:39,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:39,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:39,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,589 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,589 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,589 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,589 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:39,589 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:39,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288999641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288999645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288999648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,689 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:39,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:39,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:39,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:39,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:39,692 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:39,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:39,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:39,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:39,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,694 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:39,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:39,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:39,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:39,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:39,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:39,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:39,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:39,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:39,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:39,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:39,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:39,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:39,695 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:39,695 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:39,696 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:39,696 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:39,696 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:39,696 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:39,696 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,696 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:39,696 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:39,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:39,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5716bb3c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:39,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:39,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,700 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:39,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,700 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:39,700 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:39,700 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:39,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:39,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:39,702 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:39,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:39,705 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:39,705 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:39,705 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:39,705 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:39,705 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:39,706 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,706 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,706 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:39,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,708 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:39,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:39,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:39,711 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:39,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:39,711 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:39,711 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,711 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:39,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:39,712 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:39,712 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:39,713 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,713 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:39,814 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:39,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:39,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:39,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:39,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:39,826 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:39,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:39,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:39,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:39,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:39,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:39,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:39,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:39,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:39,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:39,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:39,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:39,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:39,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:39,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:39,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:39,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:39,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:39,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:39,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:39,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:39,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:39,831 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,831 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:39,831 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:39,832 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,832 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,832 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,833 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@610e09e1[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:39,835 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:39,835 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:39,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:39,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:39,837 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,837 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:39,838 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,838 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:39,838 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,838 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:39,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:39,840 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:39,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:39,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:39,841 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:39,841 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:39,842 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,842 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,842 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,842 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,845 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:39,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:39,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,848 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:39,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:39,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:39,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:39,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:39,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:39,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,848 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:39,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:39,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:39,848 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:39,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680288999848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680288999852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:39,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680288999857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,924 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=544, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/8186e2964c194d0f896ac5cdd85c6b21 2023-03-31 18:55:39,948 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:39,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:39,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:39,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:39,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:39,951 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:39,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:39,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:39,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:39,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,953 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:39,953 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:39,953 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,953 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:39,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:39,954 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:39,954 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:39,955 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:39,955 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:39,957 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,957 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,957 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:39,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f1f4d54[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:39,959 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:39,959 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,960 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,960 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:39,960 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:39,960 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:39,960 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,960 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:39,961 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,961 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,961 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:39,961 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:39,963 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:39,963 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:39,964 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:39,964 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:39,964 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:39,974 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,974 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,974 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:39,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:39,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:39,975 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:39,978 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:39,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:39,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:39,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:39,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:39,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:39,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:39,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:39,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:39,978 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:39,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,978 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:39,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:39,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:39,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:39,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,078 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:40,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:40,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:40,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:40,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:40,082 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:40,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:40,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:40,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:40,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,082 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,083 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:40,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:40,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:40,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:40,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:40,085 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:40,085 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:40,085 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:40,085 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:40,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:40,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c1689f8[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:40,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:40,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,090 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,091 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,091 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:40,092 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:40,092 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:40,092 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,092 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:40,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:40,094 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:40,094 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:40,094 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:40,094 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:40,094 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:40,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:40,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:40,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:40,097 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,098 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,102 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:40,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:40,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:40,102 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:40,102 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:40,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:40,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680289000161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680289000161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680289000163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,202 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:40,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:40,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:40,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:40,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:40,205 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:40,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:40,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:40,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:40,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:40,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:40,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:40,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,208 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:40,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:40,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:40,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:40,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:40,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:40,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:40,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:40,209 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:40,209 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:40,209 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:40,209 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:40,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:40,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:40,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:40,209 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:40,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:40,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:40,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e3b1a1d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:40,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:40,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:40,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:40,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:40,214 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:40,216 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:40,216 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:40,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,216 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:40,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:40,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:40,219 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:40,219 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:40,219 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:40,219 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:40,219 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:40,221 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:40,221 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,221 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,221 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:40,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:40,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:40,222 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:40,224 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:40,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:40,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:40,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:40,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,224 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,224 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:40,325 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:40,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:40,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:40,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:40,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:40,329 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:40,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:40,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:40,329 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:40,330 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,330 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,331 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:40,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:40,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:40,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:40,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:40,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:40,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:40,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:40,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:40,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:40,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:40,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:40,333 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:40,333 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:40,333 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:40,333 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:40,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:40,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:40,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:40,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:40,335 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/d3b5f2fac12644dc82df972465f84e1d 2023-03-31 18:55:40,337 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:40,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:40,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:40,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4feb413a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:40,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:40,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,342 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:40,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:40,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:40,343 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,343 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:40,344 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:40,344 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:40,344 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:40,344 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:40,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:40,349 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:40,349 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:40,349 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:40,349 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:40,349 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:40,351 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,351 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:40,351 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:40,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,352 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:40,354 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,360 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:40,360 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:40,360 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:40,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:40,360 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:40,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:40,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:40,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:40,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:40,360 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,360 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,361 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,361 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,361 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:40,361 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,361 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:40,361 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:40,362 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/961e3802766d49f18ec1325ad90a705f 2023-03-31 18:55:40,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/8186e2964c194d0f896ac5cdd85c6b21 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/8186e2964c194d0f896ac5cdd85c6b21 2023-03-31 18:55:40,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/8186e2964c194d0f896ac5cdd85c6b21, entries=250, sequenceid=544, filesize=39.2 K 2023-03-31 18:55:40,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/d3b5f2fac12644dc82df972465f84e1d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d3b5f2fac12644dc82df972465f84e1d 2023-03-31 18:55:40,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d3b5f2fac12644dc82df972465f84e1d, entries=150, sequenceid=544, filesize=11.9 K 2023-03-31 18:55:40,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/961e3802766d49f18ec1325ad90a705f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/961e3802766d49f18ec1325ad90a705f 2023-03-31 18:55:40,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/961e3802766d49f18ec1325ad90a705f, entries=150, sequenceid=544, filesize=11.9 K 2023-03-31 18:55:40,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~64.85 KB/66410, heapSize ~184.05 KB/188464, currentSize=114.05 KB/116790 for c2adb1ceb55b533a282bcfb4ded6271e in 884ms, sequenceid=544, compaction requested=true 2023-03-31 18:55:40,377 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:40,377 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:40,377 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 18:55:40,377 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 18:55:40,377 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:40,377 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:40,379 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163926 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 18:55:40,379 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71286 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 18:55:40,379 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/A is initiating minor compaction (all files) 2023-03-31 18:55:40,379 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/B is initiating minor compaction (all files) 2023-03-31 18:55:40,379 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/A in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:40,379 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/B in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:40,379 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/cbd9ffc0fa5047878ab64ba7608e7349, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a5b7074d3e4645249d1918cc70adffd2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/114d91339c334f1eadb7750464767061, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/129d55b0252d4773a8c5d61a9ee6eadc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/231edcdadfe44a14be9a4df20f1af062, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/8186e2964c194d0f896ac5cdd85c6b21] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=160.1 K 2023-03-31 18:55:40,379 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/ba5df1902a22468aab2a31e7ffd3f02a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c1e8fbeee5b44e2a8bc9d6434e924ae, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1466889453484929a4a351b2f41bfd84, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a44e0954c9724ad89d00a65ab14a47d0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/6e1f49d272f24c12b84f8cc5c63137cf, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d3b5f2fac12644dc82df972465f84e1d] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=69.6 K 2023-03-31 18:55:40,380 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting cbd9ffc0fa5047878ab64ba7608e7349, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680288933558 2023-03-31 18:55:40,380 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ba5df1902a22468aab2a31e7ffd3f02a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680288933558 2023-03-31 18:55:40,380 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a5b7074d3e4645249d1918cc70adffd2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680288934229 2023-03-31 18:55:40,380 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4c1e8fbeee5b44e2a8bc9d6434e924ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680288934229 2023-03-31 18:55:40,380 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 114d91339c334f1eadb7750464767061, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1680288934883 2023-03-31 18:55:40,380 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 1466889453484929a4a351b2f41bfd84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1680288934883 2023-03-31 18:55:40,381 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 129d55b0252d4773a8c5d61a9ee6eadc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680288936034 2023-03-31 18:55:40,381 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a44e0954c9724ad89d00a65ab14a47d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680288936034 2023-03-31 18:55:40,381 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 231edcdadfe44a14be9a4df20f1af062, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1680288938185 2023-03-31 18:55:40,381 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 6e1f49d272f24c12b84f8cc5c63137cf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1680288938185 2023-03-31 18:55:40,381 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8186e2964c194d0f896ac5cdd85c6b21, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680288939371 2023-03-31 18:55:40,381 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d3b5f2fac12644dc82df972465f84e1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680288939371 2023-03-31 18:55:40,392 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#A#compaction#357 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:40,393 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#B#compaction#358 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:40,401 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/e83fc0aa0fe444bdb8e96a8210e973fd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/e83fc0aa0fe444bdb8e96a8210e973fd 2023-03-31 18:55:40,405 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/B of c2adb1ceb55b533a282bcfb4ded6271e into e83fc0aa0fe444bdb8e96a8210e973fd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:40,405 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:40,405 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/B, priority=10, startTime=1680288940377; duration=0sec 2023-03-31 18:55:40,405 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:40,405 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 18:55:40,406 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71286 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 18:55:40,406 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/C is initiating minor compaction (all files) 2023-03-31 18:55:40,406 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/C in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:40,406 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e5b6c4ce2f8241c98d36503c1fd28df4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/2155a03e8b3f46cbb72aa33cd7b76092, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91d71d74c6904e2e8ab77c90d8a43523, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/68550689c934462cbc61a6a489776daf, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/1001a91ffc844301a69c54beebb9bbba, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/961e3802766d49f18ec1325ad90a705f] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=69.6 K 2023-03-31 18:55:40,407 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e5b6c4ce2f8241c98d36503c1fd28df4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680288933558 2023-03-31 18:55:40,407 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2155a03e8b3f46cbb72aa33cd7b76092, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680288934229 2023-03-31 18:55:40,407 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 91d71d74c6904e2e8ab77c90d8a43523, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1680288934883 2023-03-31 18:55:40,407 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 68550689c934462cbc61a6a489776daf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680288936034 2023-03-31 18:55:40,408 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 1001a91ffc844301a69c54beebb9bbba, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1680288938185 2023-03-31 18:55:40,408 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 961e3802766d49f18ec1325ad90a705f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680288939371 2023-03-31 18:55:40,417 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#C#compaction#359 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:40,424 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/9593fcb0b5fb49a4b12399095e203afe as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9593fcb0b5fb49a4b12399095e203afe 2023-03-31 18:55:40,427 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/C of c2adb1ceb55b533a282bcfb4ded6271e into 9593fcb0b5fb49a4b12399095e203afe(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:40,427 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:40,427 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/C, priority=10, startTime=1680288940377; duration=0sec 2023-03-31 18:55:40,428 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:40,461 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:40,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:40,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:40,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:40,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:40,464 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:40,464 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:40,464 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:40,464 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:40,464 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,464 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,466 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:40,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:40,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:40,466 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,466 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:40,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:40,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:40,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:40,467 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:40,467 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:40,467 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:40,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:40,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:40,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:40,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:40,468 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:55:40,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:40,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:40,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:40,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:40,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:40,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:40,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:40,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:40,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680289000702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680289000704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680289000705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47904 deadline: 1680289000736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,740 DEBUG [Thread-2248] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18277 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:40,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1680289000746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,749 DEBUG [Thread-2254] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18314 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:40,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680289000806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680289000813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:40,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680289000813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:40,881 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303317b2c2e41c7c045ddae8577701408d5a3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317b2c2e41c7c045ddae8577701408d5a3 2023-03-31 18:55:40,887 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=566, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/9f489c68052d4f9a8bdd36a8b010d190 2023-03-31 18:55:40,910 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/0191c0a3134e42778cfb18f8f6f19890 2023-03-31 18:55:41,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680289001010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680289001016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680289001016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,208 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/a618448607194311a4bdc5faa59f6cf7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a618448607194311a4bdc5faa59f6cf7 2023-03-31 18:55:41,212 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/A of c2adb1ceb55b533a282bcfb4ded6271e into a618448607194311a4bdc5faa59f6cf7(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:41,212 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:41,212 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/A, priority=10, startTime=1680288940377; duration=0sec 2023-03-31 18:55:41,213 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:41,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680289001311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680289001322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680289001322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,339 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/f9045b3ec6a34961a4cff955fa9d3029 2023-03-31 18:55:41,343 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/9f489c68052d4f9a8bdd36a8b010d190 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9f489c68052d4f9a8bdd36a8b010d190 2023-03-31 18:55:41,346 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9f489c68052d4f9a8bdd36a8b010d190, entries=150, sequenceid=566, filesize=25.4 K 2023-03-31 18:55:41,347 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/0191c0a3134e42778cfb18f8f6f19890 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0191c0a3134e42778cfb18f8f6f19890 2023-03-31 18:55:41,350 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0191c0a3134e42778cfb18f8f6f19890, entries=150, sequenceid=566, filesize=11.9 K 2023-03-31 18:55:41,351 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/f9045b3ec6a34961a4cff955fa9d3029 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/f9045b3ec6a34961a4cff955fa9d3029 2023-03-31 18:55:41,355 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/f9045b3ec6a34961a4cff955fa9d3029, entries=150, sequenceid=566, filesize=11.9 K 2023-03-31 18:55:41,355 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c2adb1ceb55b533a282bcfb4ded6271e in 887ms, sequenceid=566, compaction requested=false 2023-03-31 18:55:41,355 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:41,356 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:41,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:41,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:41,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:41,356 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:41,357 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,357 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:41,358 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:41,358 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a0bd07e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:41,359 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:41,359 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:41,360 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:41,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:41,360 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:41,360 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:41,361 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:41,361 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,361 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:41,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,362 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:41,362 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:41,362 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:41,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:41,365 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:41,365 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:41,365 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:41,365 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:41,365 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:41,368 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:41,368 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:41,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:41,368 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:41,368 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:41,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:41,370 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:41,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:41,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:41,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:41,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:41,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:41,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:41,373 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,373 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:41,373 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:41,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:41,373 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:41,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:41,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:41,380 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:41,380 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:41,480 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:41,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:41,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:41,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:41,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:41,484 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:41,484 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:41,484 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:41,484 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:41,484 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:41,485 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:41,486 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:41,486 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:41,486 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,486 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:41,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:41,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:41,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:41,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:41,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:41,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:41,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:41,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:41,488 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:41,488 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:41,488 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. started... 2023-03-31 18:55:41,489 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:55:41,489 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:41,489 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:41,489 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:41,489 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:41,489 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:41,489 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:41,513 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331d7caaa675101460eb460aba04072c0c9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d7caaa675101460eb460aba04072c0c9 2023-03-31 18:55:41,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:41,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. as already flushing 2023-03-31 18:55:41,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680289001872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680289001876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680289001877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,935 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=579, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/92ad51aaffe74c1a8f8ea479f3c4c248 2023-03-31 18:55:41,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680289001979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680289001985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:41,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:41,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680289001985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:42,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47844 deadline: 1680289002186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:42,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47856 deadline: 1680289002190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:42,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47870 deadline: 1680289002190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,348 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/a2e745a6dd684d1f8fb90c5afdc4965a 2023-03-31 18:55:42,358 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/73b399c666634922ae032cca262256db 2023-03-31 18:55:42,361 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/92ad51aaffe74c1a8f8ea479f3c4c248 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/92ad51aaffe74c1a8f8ea479f3c4c248 2023-03-31 18:55:42,364 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/92ad51aaffe74c1a8f8ea479f3c4c248, entries=100, sequenceid=579, filesize=18.6 K 2023-03-31 18:55:42,365 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/a2e745a6dd684d1f8fb90c5afdc4965a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a2e745a6dd684d1f8fb90c5afdc4965a 2023-03-31 18:55:42,366 DEBUG [Thread-2257] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:63858 2023-03-31 18:55:42,366 DEBUG [Thread-2257] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:42,368 DEBUG [Thread-2259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:63858 2023-03-31 18:55:42,368 DEBUG [Thread-2259] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:42,369 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a2e745a6dd684d1f8fb90c5afdc4965a, entries=100, sequenceid=579, filesize=9.5 K 2023-03-31 18:55:42,370 DEBUG [Thread-2261] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:63858 2023-03-31 18:55:42,370 DEBUG [Thread-2261] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:42,370 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/73b399c666634922ae032cca262256db as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/73b399c666634922ae032cca262256db 2023-03-31 18:55:42,373 DEBUG [Thread-2265] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a7503d4 to 127.0.0.1:63858 2023-03-31 18:55:42,373 DEBUG [Thread-2265] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:42,374 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/73b399c666634922ae032cca262256db, entries=100, sequenceid=579, filesize=9.5 K 2023-03-31 18:55:42,375 DEBUG [Thread-2263] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x711ebaf2 to 127.0.0.1:63858 2023-03-31 18:55:42,375 DEBUG [Thread-2263] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:42,375 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c2adb1ceb55b533a282bcfb4ded6271e in 886ms, sequenceid=579, compaction requested=true 2023-03-31 18:55:42,375 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:42,375 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:42,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:42,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:42,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:42,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:42,378 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:42,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:42,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:42,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52dadb8e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:42,379 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:42,379 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:42,381 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:42,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:42,381 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:42,381 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:42,381 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:42,381 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,381 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:42,382 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:42,382 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,382 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:42,382 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:42,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:42,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:42,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:42,384 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:42,384 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:42,384 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:42,384 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:42,384 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:42,386 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:42,386 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:42,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:42,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:42,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:42,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:42,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:42,386 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:42,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:42,386 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:42,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:42,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:42,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:42,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:42,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:42,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,387 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:42,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:42,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:42,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:42,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:42,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:42,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:42,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:42,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:42,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:42,389 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:42,390 DEBUG [Thread-2256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:42,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:42,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:42,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:42,390 DEBUG [Thread-2256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:42,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:42,390 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:42,490 DEBUG [Thread-2256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:42,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:42,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:42,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-03-31 18:55:42,493 DEBUG [Thread-2246] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:63858 2023-03-31 18:55:42,493 DEBUG [Thread-2250] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:63858 2023-03-31 18:55:42,493 DEBUG [Thread-2246] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:42,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:42,493 DEBUG [Thread-2250] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:42,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:42,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:42,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:42,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:42,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:42,494 DEBUG [Thread-2252] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69100738 to 127.0.0.1:63858 2023-03-31 18:55:42,495 DEBUG [Thread-2252] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:42,904 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331b617a08a8127437da2a67fb155970d2a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b617a08a8127437da2a67fb155970d2a 2023-03-31 18:55:43,309 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=601, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/9ef54cb9c28145e6b2643c02532c7274 2023-03-31 18:55:43,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/cffc62964e0e4fc9bf317df96a4e39ef 2023-03-31 18:55:44,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/edcb189e67ed43d482b9684d218d6d05 2023-03-31 18:55:44,131 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/9ef54cb9c28145e6b2643c02532c7274 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ef54cb9c28145e6b2643c02532c7274 2023-03-31 18:55:44,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ef54cb9c28145e6b2643c02532c7274, entries=150, sequenceid=601, filesize=25.4 K 2023-03-31 18:55:44,134 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/cffc62964e0e4fc9bf317df96a4e39ef as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/cffc62964e0e4fc9bf317df96a4e39ef 2023-03-31 18:55:44,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/cffc62964e0e4fc9bf317df96a4e39ef, entries=150, sequenceid=601, filesize=11.9 K 2023-03-31 18:55:44,137 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/edcb189e67ed43d482b9684d218d6d05 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/edcb189e67ed43d482b9684d218d6d05 2023-03-31 18:55:44,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/edcb189e67ed43d482b9684d218d6d05, entries=150, sequenceid=601, filesize=11.9 K 2023-03-31 18:55:44,141 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=6.71 KB/6870 for c2adb1ceb55b533a282bcfb4ded6271e in 1648ms, sequenceid=601, compaction requested=true 2023-03-31 18:55:44,141 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:44,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:44,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:44,141 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:44,141 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:44,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:44,142 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97342 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:44,142 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46506 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:44,142 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/A is initiating minor compaction (all files) 2023-03-31 18:55:44,142 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/B is initiating minor compaction (all files) 2023-03-31 18:55:44,142 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/A in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:44,142 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/B in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:44,142 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a618448607194311a4bdc5faa59f6cf7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9f489c68052d4f9a8bdd36a8b010d190, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/92ad51aaffe74c1a8f8ea479f3c4c248, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ef54cb9c28145e6b2643c02532c7274] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=95.1 K 2023-03-31 18:55:44,142 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/e83fc0aa0fe444bdb8e96a8210e973fd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0191c0a3134e42778cfb18f8f6f19890, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a2e745a6dd684d1f8fb90c5afdc4965a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/cffc62964e0e4fc9bf317df96a4e39ef] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=45.4 K 2023-03-31 18:55:44,142 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a618448607194311a4bdc5faa59f6cf7, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680288939371 2023-03-31 18:55:44,142 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e83fc0aa0fe444bdb8e96a8210e973fd, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680288939371 2023-03-31 18:55:44,143 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9f489c68052d4f9a8bdd36a8b010d190, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1680288939526 2023-03-31 18:55:44,143 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 0191c0a3134e42778cfb18f8f6f19890, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1680288939526 2023-03-31 18:55:44,143 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 92ad51aaffe74c1a8f8ea479f3c4c248, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1680288940695 2023-03-31 18:55:44,143 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a2e745a6dd684d1f8fb90c5afdc4965a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1680288940695 2023-03-31 18:55:44,143 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9ef54cb9c28145e6b2643c02532c7274, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680288941853 2023-03-31 18:55:44,143 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting cffc62964e0e4fc9bf317df96a4e39ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680288941853 2023-03-31 18:55:44,149 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#B#compaction#370 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:44,151 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#A#compaction#369 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:44,558 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/dea47f0383af459cb084a0ebf989ef6d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/dea47f0383af459cb084a0ebf989ef6d 2023-03-31 18:55:44,562 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/B of c2adb1ceb55b533a282bcfb4ded6271e into dea47f0383af459cb084a0ebf989ef6d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:44,562 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:44,562 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/B, priority=12, startTime=1680288944141; duration=0sec 2023-03-31 18:55:44,562 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:44,562 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:44,563 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46506 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:44,563 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): c2adb1ceb55b533a282bcfb4ded6271e/C is initiating minor compaction (all files) 2023-03-31 18:55:44,563 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2adb1ceb55b533a282bcfb4ded6271e/C in TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:44,563 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9593fcb0b5fb49a4b12399095e203afe, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/f9045b3ec6a34961a4cff955fa9d3029, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/73b399c666634922ae032cca262256db, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/edcb189e67ed43d482b9684d218d6d05] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp, totalSize=45.4 K 2023-03-31 18:55:44,564 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9593fcb0b5fb49a4b12399095e203afe, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680288939371 2023-03-31 18:55:44,564 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f9045b3ec6a34961a4cff955fa9d3029, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1680288939526 2023-03-31 18:55:44,564 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 73b399c666634922ae032cca262256db, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1680288940695 2023-03-31 18:55:44,565 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting edcb189e67ed43d482b9684d218d6d05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680288941853 2023-03-31 18:55:44,571 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): c2adb1ceb55b533a282bcfb4ded6271e#C#compaction#371 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:44,977 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/929ed59dcdf1447cb7070e97811a0833 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/929ed59dcdf1447cb7070e97811a0833 2023-03-31 18:55:44,980 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/C of c2adb1ceb55b533a282bcfb4ded6271e into 929ed59dcdf1447cb7070e97811a0833(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:44,981 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:44,981 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/C, priority=12, startTime=1680288944141; duration=0sec 2023-03-31 18:55:44,981 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:45,367 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/870106a93b4a47ce916e0333bf383c62 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/870106a93b4a47ce916e0333bf383c62 2023-03-31 18:55:45,370 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2adb1ceb55b533a282bcfb4ded6271e/A of c2adb1ceb55b533a282bcfb4ded6271e into 870106a93b4a47ce916e0333bf383c62(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-31 18:55:45,370 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:45,370 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e., storeName=c2adb1ceb55b533a282bcfb4ded6271e/A, priority=12, startTime=1680288944141; duration=1sec 2023-03-31 18:55:45,370 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:50,764 DEBUG [Thread-2248] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:63858 2023-03-31 18:55:50,764 DEBUG [Thread-2248] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:50,810 DEBUG [Thread-2254] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f603c27 to 127.0.0.1:63858 2023-03-31 18:55:50,811 DEBUG [Thread-2254] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 149 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 168 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 156 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 13 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1726 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5176 rows 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1732 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5196 rows 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1735 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5204 rows 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1739 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5194 rows 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1739 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5217 rows 2023-03-31 18:55:50,811 INFO [Listener at localhost/43327] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 18:55:50,811 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6434ce29 to 127.0.0.1:63858 2023-03-31 18:55:50,812 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:55:50,815 INFO [Listener at localhost/43327] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 18:55:50,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 18:55:50,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:50,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 18:55:50,819 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288950819"}]},"ts":"1680288950819"} 2023-03-31 18:55:50,820 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 18:55:50,823 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 18:55:50,823 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, UNASSIGN}] 2023-03-31 18:55:50,824 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, UNASSIGN 2023-03-31 18:55:50,825 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=c2adb1ceb55b533a282bcfb4ded6271e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:50,825 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288950825"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288950825"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288950825"}]},"ts":"1680288950825"} 2023-03-31 18:55:50,826 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:55:50,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 18:55:50,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:50,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c2adb1ceb55b533a282bcfb4ded6271e, disabling compactions & flushes 2023-03-31 18:55:50,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:50,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:50,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. after waiting 0 ms 2023-03-31 18:55:50,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:50,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c2adb1ceb55b533a282bcfb4ded6271e 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-31 18:55:50,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=A 2023-03-31 18:55:50,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:50,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=B 2023-03-31 18:55:50,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:50,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2adb1ceb55b533a282bcfb4ded6271e, store=C 2023-03-31 18:55:50,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:51,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 18:55:51,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331d9a07e4e2a1b461480f7571fd96bd8c7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d9a07e4e2a1b461480f7571fd96bd8c7 2023-03-31 18:55:51,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 18:55:51,794 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=610, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/26c326836fee4da699b06c407f2e9dd9 2023-03-31 18:55:51,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 18:55:52,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/8611c572c27140f184df1f71cb1b4415 2023-03-31 18:55:52,611 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/53fbb01800f64e5caf5eb31f16f59dc3 2023-03-31 18:55:52,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/A/26c326836fee4da699b06c407f2e9dd9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/26c326836fee4da699b06c407f2e9dd9 2023-03-31 18:55:52,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/26c326836fee4da699b06c407f2e9dd9, entries=100, sequenceid=610, filesize=18.6 K 2023-03-31 18:55:52,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/B/8611c572c27140f184df1f71cb1b4415 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/8611c572c27140f184df1f71cb1b4415 2023-03-31 18:55:52,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/8611c572c27140f184df1f71cb1b4415, entries=100, sequenceid=610, filesize=9.5 K 2023-03-31 18:55:52,628 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/.tmp/C/53fbb01800f64e5caf5eb31f16f59dc3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/53fbb01800f64e5caf5eb31f16f59dc3 2023-03-31 18:55:52,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/53fbb01800f64e5caf5eb31f16f59dc3, entries=100, sequenceid=610, filesize=9.5 K 2023-03-31 18:55:52,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for c2adb1ceb55b533a282bcfb4ded6271e in 1655ms, sequenceid=610, compaction requested=false 2023-03-31 18:55:52,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/cbd9ffc0fa5047878ab64ba7608e7349, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a5b7074d3e4645249d1918cc70adffd2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/114d91339c334f1eadb7750464767061, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/129d55b0252d4773a8c5d61a9ee6eadc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/231edcdadfe44a14be9a4df20f1af062, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/8186e2964c194d0f896ac5cdd85c6b21, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a618448607194311a4bdc5faa59f6cf7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9f489c68052d4f9a8bdd36a8b010d190, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/92ad51aaffe74c1a8f8ea479f3c4c248, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ef54cb9c28145e6b2643c02532c7274] to archive 2023-03-31 18:55:52,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:55:52,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/cbd9ffc0fa5047878ab64ba7608e7349 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/cbd9ffc0fa5047878ab64ba7608e7349 2023-03-31 18:55:52,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a5b7074d3e4645249d1918cc70adffd2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a5b7074d3e4645249d1918cc70adffd2 2023-03-31 18:55:52,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/114d91339c334f1eadb7750464767061 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/114d91339c334f1eadb7750464767061 2023-03-31 18:55:52,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/129d55b0252d4773a8c5d61a9ee6eadc to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/129d55b0252d4773a8c5d61a9ee6eadc 2023-03-31 18:55:52,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/231edcdadfe44a14be9a4df20f1af062 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/231edcdadfe44a14be9a4df20f1af062 2023-03-31 18:55:52,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/8186e2964c194d0f896ac5cdd85c6b21 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/8186e2964c194d0f896ac5cdd85c6b21 2023-03-31 18:55:52,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a618448607194311a4bdc5faa59f6cf7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/a618448607194311a4bdc5faa59f6cf7 2023-03-31 18:55:52,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9f489c68052d4f9a8bdd36a8b010d190 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9f489c68052d4f9a8bdd36a8b010d190 2023-03-31 18:55:52,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/92ad51aaffe74c1a8f8ea479f3c4c248 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/92ad51aaffe74c1a8f8ea479f3c4c248 2023-03-31 18:55:52,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ef54cb9c28145e6b2643c02532c7274 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/9ef54cb9c28145e6b2643c02532c7274 2023-03-31 18:55:52,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/ba5df1902a22468aab2a31e7ffd3f02a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c1e8fbeee5b44e2a8bc9d6434e924ae, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1466889453484929a4a351b2f41bfd84, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a44e0954c9724ad89d00a65ab14a47d0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/6e1f49d272f24c12b84f8cc5c63137cf, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/e83fc0aa0fe444bdb8e96a8210e973fd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d3b5f2fac12644dc82df972465f84e1d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0191c0a3134e42778cfb18f8f6f19890, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a2e745a6dd684d1f8fb90c5afdc4965a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/cffc62964e0e4fc9bf317df96a4e39ef] to archive 2023-03-31 18:55:52,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:55:52,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/ba5df1902a22468aab2a31e7ffd3f02a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/ba5df1902a22468aab2a31e7ffd3f02a 2023-03-31 18:55:52,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c1e8fbeee5b44e2a8bc9d6434e924ae to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/4c1e8fbeee5b44e2a8bc9d6434e924ae 2023-03-31 18:55:52,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1466889453484929a4a351b2f41bfd84 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/1466889453484929a4a351b2f41bfd84 2023-03-31 18:55:52,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a44e0954c9724ad89d00a65ab14a47d0 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a44e0954c9724ad89d00a65ab14a47d0 2023-03-31 18:55:52,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/6e1f49d272f24c12b84f8cc5c63137cf to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/6e1f49d272f24c12b84f8cc5c63137cf 2023-03-31 18:55:52,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/e83fc0aa0fe444bdb8e96a8210e973fd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/e83fc0aa0fe444bdb8e96a8210e973fd 2023-03-31 18:55:52,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d3b5f2fac12644dc82df972465f84e1d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/d3b5f2fac12644dc82df972465f84e1d 2023-03-31 18:55:52,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0191c0a3134e42778cfb18f8f6f19890 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/0191c0a3134e42778cfb18f8f6f19890 2023-03-31 18:55:52,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a2e745a6dd684d1f8fb90c5afdc4965a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/a2e745a6dd684d1f8fb90c5afdc4965a 2023-03-31 18:55:52,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/cffc62964e0e4fc9bf317df96a4e39ef to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/cffc62964e0e4fc9bf317df96a4e39ef 2023-03-31 18:55:52,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e5b6c4ce2f8241c98d36503c1fd28df4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/2155a03e8b3f46cbb72aa33cd7b76092, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91d71d74c6904e2e8ab77c90d8a43523, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/68550689c934462cbc61a6a489776daf, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/1001a91ffc844301a69c54beebb9bbba, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9593fcb0b5fb49a4b12399095e203afe, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/961e3802766d49f18ec1325ad90a705f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/f9045b3ec6a34961a4cff955fa9d3029, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/73b399c666634922ae032cca262256db, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/edcb189e67ed43d482b9684d218d6d05] to archive 2023-03-31 18:55:52,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:55:52,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e5b6c4ce2f8241c98d36503c1fd28df4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/e5b6c4ce2f8241c98d36503c1fd28df4 2023-03-31 18:55:52,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/2155a03e8b3f46cbb72aa33cd7b76092 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/2155a03e8b3f46cbb72aa33cd7b76092 2023-03-31 18:55:52,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91d71d74c6904e2e8ab77c90d8a43523 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/91d71d74c6904e2e8ab77c90d8a43523 2023-03-31 18:55:52,675 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/68550689c934462cbc61a6a489776daf to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/68550689c934462cbc61a6a489776daf 2023-03-31 18:55:52,676 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/1001a91ffc844301a69c54beebb9bbba to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/1001a91ffc844301a69c54beebb9bbba 2023-03-31 18:55:52,680 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9593fcb0b5fb49a4b12399095e203afe to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/9593fcb0b5fb49a4b12399095e203afe 2023-03-31 18:55:52,681 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/961e3802766d49f18ec1325ad90a705f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/961e3802766d49f18ec1325ad90a705f 2023-03-31 18:55:52,682 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/f9045b3ec6a34961a4cff955fa9d3029 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/f9045b3ec6a34961a4cff955fa9d3029 2023-03-31 18:55:52,683 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/73b399c666634922ae032cca262256db to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/73b399c666634922ae032cca262256db 2023-03-31 18:55:52,684 DEBUG [StoreCloser-TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/edcb189e67ed43d482b9684d218d6d05 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/edcb189e67ed43d482b9684d218d6d05 2023-03-31 18:55:52,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/recovered.edits/613.seqid, newMaxSeqId=613, maxSeqId=4 2023-03-31 18:55:52,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e. 2023-03-31 18:55:52,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c2adb1ceb55b533a282bcfb4ded6271e: 2023-03-31 18:55:52,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:52,691 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=c2adb1ceb55b533a282bcfb4ded6271e, regionState=CLOSED 2023-03-31 18:55:52,691 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288952691"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288952691"}]},"ts":"1680288952691"} 2023-03-31 18:55:52,693 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-31 18:55:52,693 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure c2adb1ceb55b533a282bcfb4ded6271e, server=jenkins-hbase4.apache.org,34951,1680288816339 in 1.8660 sec 2023-03-31 18:55:52,695 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-31 18:55:52,695 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2adb1ceb55b533a282bcfb4ded6271e, UNASSIGN in 1.8700 sec 2023-03-31 18:55:52,695 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288952695"}]},"ts":"1680288952695"} 2023-03-31 18:55:52,696 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 18:55:52,697 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 18:55:52,699 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8830 sec 2023-03-31 18:55:52,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 18:55:52,923 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-31 18:55:52,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 18:55:52,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:52,925 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:52,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-31 18:55:52,926 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:52,930 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:52,932 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/recovered.edits] 2023-03-31 18:55:52,934 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/26c326836fee4da699b06c407f2e9dd9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/26c326836fee4da699b06c407f2e9dd9 2023-03-31 18:55:52,935 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/870106a93b4a47ce916e0333bf383c62 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/A/870106a93b4a47ce916e0333bf383c62 2023-03-31 18:55:52,937 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/8611c572c27140f184df1f71cb1b4415 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/8611c572c27140f184df1f71cb1b4415 2023-03-31 18:55:52,938 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/dea47f0383af459cb084a0ebf989ef6d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/B/dea47f0383af459cb084a0ebf989ef6d 2023-03-31 18:55:52,939 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/53fbb01800f64e5caf5eb31f16f59dc3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/53fbb01800f64e5caf5eb31f16f59dc3 2023-03-31 18:55:52,940 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/929ed59dcdf1447cb7070e97811a0833 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/C/929ed59dcdf1447cb7070e97811a0833 2023-03-31 18:55:52,943 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/recovered.edits/613.seqid to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e/recovered.edits/613.seqid 2023-03-31 18:55:52,943 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/c2adb1ceb55b533a282bcfb4ded6271e 2023-03-31 18:55:52,943 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:55:52,943 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 18:55:52,944 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-31 18:55:52,947 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331397a4923eee5460b9f7c1dec4e8cf233 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331397a4923eee5460b9f7c1dec4e8cf233 2023-03-31 18:55:52,947 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033140fd5f2dbdb449609c20885703c2469d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033140fd5f2dbdb449609c20885703c2469d 2023-03-31 18:55:52,948 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303314d968589d1484bc6ae07126cfc9cc0e2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303314d968589d1484bc6ae07126cfc9cc0e2 2023-03-31 18:55:52,949 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331512af9348fba4262b8dcda00fd2ff11f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331512af9348fba4262b8dcda00fd2ff11f 2023-03-31 18:55:52,950 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033152cb769ec77a44a6ad0b5a12c0fd58e0 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033152cb769ec77a44a6ad0b5a12c0fd58e0 2023-03-31 18:55:52,951 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033154f59e56a64a4ab795c00d2b30bd26cf to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033154f59e56a64a4ab795c00d2b30bd26cf 2023-03-31 18:55:52,952 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303316af901a6119e4cfaa8d600ac95280a0d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303316af901a6119e4cfaa8d600ac95280a0d 2023-03-31 18:55:52,952 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033173e48f2d8b814ee7bba044c793221539 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033173e48f2d8b814ee7bba044c793221539 2023-03-31 18:55:52,953 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317b2c2e41c7c045ddae8577701408d5a3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317b2c2e41c7c045ddae8577701408d5a3 2023-03-31 18:55:52,954 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318607d48a08a6482b8caf15a51b40e9c4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318607d48a08a6482b8caf15a51b40e9c4 2023-03-31 18:55:52,955 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319d74973cc56646548b223f78ffbf675b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319d74973cc56646548b223f78ffbf675b 2023-03-31 18:55:52,956 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331aa0c0226dc1d4f68a95bb476319458c4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331aa0c0226dc1d4f68a95bb476319458c4 2023-03-31 18:55:52,957 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b617a08a8127437da2a67fb155970d2a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b617a08a8127437da2a67fb155970d2a 2023-03-31 18:55:52,958 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b92dfbb9fa9146f0882a9ecbfcd179b9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b92dfbb9fa9146f0882a9ecbfcd179b9 2023-03-31 18:55:52,958 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331bd648342603c43fd9267afa7605ce884 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331bd648342603c43fd9267afa7605ce884 2023-03-31 18:55:52,959 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331cba6944e19e5493daff42f78e63381bd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331cba6944e19e5493daff42f78e63381bd 2023-03-31 18:55:52,960 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d7caaa675101460eb460aba04072c0c9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d7caaa675101460eb460aba04072c0c9 2023-03-31 18:55:52,961 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d9a07e4e2a1b461480f7571fd96bd8c7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d9a07e4e2a1b461480f7571fd96bd8c7 2023-03-31 18:55:52,962 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331dac63a512b5d40c88e8a1d1948cd6d4f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331dac63a512b5d40c88e8a1d1948cd6d4f 2023-03-31 18:55:52,962 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e1b01804f59248dd9819ebb8259b1b4b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e1b01804f59248dd9819ebb8259b1b4b 2023-03-31 18:55:52,963 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e7a67f988f79445ebf285b7e7891f4f5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e7a67f988f79445ebf285b7e7891f4f5 2023-03-31 18:55:52,964 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ef2610791ca14ebc8a578ab0e5ae1473 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ef2610791ca14ebc8a578ab0e5ae1473 2023-03-31 18:55:52,965 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ff218b660519403c9af7b94fe275d230 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ff218b660519403c9af7b94fe275d230 2023-03-31 18:55:52,965 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 18:55:52,967 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:52,968 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 18:55:52,969 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 18:55:52,970 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:52,970 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 18:55:52,970 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680288952970"}]},"ts":"9223372036854775807"} 2023-03-31 18:55:52,972 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 18:55:52,972 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c2adb1ceb55b533a282bcfb4ded6271e, NAME => 'TestAcidGuarantees,,1680288919292.c2adb1ceb55b533a282bcfb4ded6271e.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 18:55:52,972 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 18:55:52,972 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680288952972"}]},"ts":"9223372036854775807"} 2023-03-31 18:55:52,973 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 18:55:52,975 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:52,976 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 52 msec 2023-03-31 18:55:53,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-31 18:55:53,027 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-31 18:55:53,038 INFO [Listener at localhost/43327] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226 (was 224) - Thread LEAK? -, OpenFileDescriptor=534 (was 566), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=296 (was 308), ProcessCount=169 (was 169), AvailableMemoryMB=3420 (was 3460) 2023-03-31 18:55:53,049 INFO [Listener at localhost/43327] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=226, OpenFileDescriptor=534, MaxFileDescriptor=60000, SystemLoadAverage=296, ProcessCount=169, AvailableMemoryMB=3419 2023-03-31 18:55:53,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 18:55:53,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 18:55:53,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 18:55:53,053 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 18:55:53,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-31 18:55:53,053 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 18:55:53,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-31 18:55:53,054 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,055 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf empty. 2023-03-31 18:55:53,055 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,055 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:55:53,062 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 18:55:53,064 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 91eb2b257640c6018ad8eeccf0deb2cf, NAME => 'TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp 2023-03-31 18:55:53,069 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:55:53,069 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 91eb2b257640c6018ad8eeccf0deb2cf, disabling compactions & flushes 2023-03-31 18:55:53,069 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,069 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,069 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. after waiting 0 ms 2023-03-31 18:55:53,069 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,069 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,069 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:53,071 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 18:55:53,072 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288953072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288953072"}]},"ts":"1680288953072"} 2023-03-31 18:55:53,073 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 18:55:53,074 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 18:55:53,074 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288953074"}]},"ts":"1680288953074"} 2023-03-31 18:55:53,075 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 18:55:53,079 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=91eb2b257640c6018ad8eeccf0deb2cf, ASSIGN}] 2023-03-31 18:55:53,080 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=91eb2b257640c6018ad8eeccf0deb2cf, ASSIGN 2023-03-31 18:55:53,081 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=91eb2b257640c6018ad8eeccf0deb2cf, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34951,1680288816339; forceNewPlan=false, retain=false 2023-03-31 18:55:53,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-31 18:55:53,232 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=91eb2b257640c6018ad8eeccf0deb2cf, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,232 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288953232"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288953232"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288953232"}]},"ts":"1680288953232"} 2023-03-31 18:55:53,233 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:55:53,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-31 18:55:53,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 91eb2b257640c6018ad8eeccf0deb2cf, NAME => 'TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:55:53,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:55:53,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,390 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,391 DEBUG [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A 2023-03-31 18:55:53,391 DEBUG [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A 2023-03-31 18:55:53,391 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:55:53,392 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 91eb2b257640c6018ad8eeccf0deb2cf columnFamilyName A 2023-03-31 18:55:53,392 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] regionserver.HStore(310): Store=91eb2b257640c6018ad8eeccf0deb2cf/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:55:53,392 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,393 DEBUG [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B 2023-03-31 18:55:53,393 DEBUG [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B 2023-03-31 18:55:53,394 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:55:53,394 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 91eb2b257640c6018ad8eeccf0deb2cf columnFamilyName B 2023-03-31 18:55:53,394 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] regionserver.HStore(310): Store=91eb2b257640c6018ad8eeccf0deb2cf/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:55:53,394 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,395 DEBUG [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C 2023-03-31 18:55:53,395 DEBUG [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C 2023-03-31 18:55:53,396 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:55:53,396 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 91eb2b257640c6018ad8eeccf0deb2cf columnFamilyName C 2023-03-31 18:55:53,396 INFO [StoreOpener-91eb2b257640c6018ad8eeccf0deb2cf-1] regionserver.HStore(310): Store=91eb2b257640c6018ad8eeccf0deb2cf/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:55:53,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:55:53,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 18:55:53,402 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 91eb2b257640c6018ad8eeccf0deb2cf; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71360838, jitterRate=0.0633593499660492}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:55:53,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:53,403 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., pid=49, masterSystemTime=1680288953385 2023-03-31 18:55:53,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,405 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=91eb2b257640c6018ad8eeccf0deb2cf, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,405 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288953404"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680288953404"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680288953404"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680288953404"}]},"ts":"1680288953404"} 2023-03-31 18:55:53,407 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-31 18:55:53,407 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 in 172 msec 2023-03-31 18:55:53,408 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-31 18:55:53,408 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=91eb2b257640c6018ad8eeccf0deb2cf, ASSIGN in 328 msec 2023-03-31 18:55:53,408 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 18:55:53,409 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288953408"}]},"ts":"1680288953408"} 2023-03-31 18:55:53,409 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 18:55:53,411 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 18:55:53,412 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 360 msec 2023-03-31 18:55:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-31 18:55:53,656 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-31 18:55:53,657 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x5e372785 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,664 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a987907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,665 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,667 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,668 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 18:55:53,669 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56286, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 18:55:53,670 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x501c5a88 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,674 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37705401, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,675 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x27f2d997 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,678 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15880ffb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,678 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,682 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,682 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,685 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,685 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x06edd017 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,689 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51b8df2a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,690 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,693 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,694 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,697 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,697 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x5f5363d8 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,702 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78592df0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,702 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x211582f2 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,707 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@659c68ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,707 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x65816b6e to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:55:53,710 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@680e773e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:55:53,714 DEBUG [hconnection-0x1614274a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,715 DEBUG [hconnection-0x43b0fe87-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,715 DEBUG [hconnection-0x6aace373-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,715 DEBUG [hconnection-0x3eefb34a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,717 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33098, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,717 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33112, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,717 DEBUG [hconnection-0x5bb64324-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,720 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33126, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,721 DEBUG [hconnection-0x4bdd68e0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,721 DEBUG [hconnection-0x7f93d9fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,721 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:53,722 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,723 DEBUG [hconnection-0x7df3d1b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,723 DEBUG [hconnection-0x5cf3fb94-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,723 DEBUG [hconnection-0x14496706-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:55:53,729 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33174, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,731 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33166, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,732 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33156, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,734 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,734 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33188, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:55:53,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:53,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:53,742 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:53,742 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:53,742 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:53,742 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:53,744 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,744 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:53,745 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,745 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:53,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:53,746 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:53,746 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:53,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:53,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:53,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:53,750 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:55:53,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:55:53,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:53,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:55:53,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:53,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:55:53,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:53,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:53,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:53,754 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,754 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:53,754 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:53,754 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,755 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:53,755 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:53,756 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,756 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:53,756 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,756 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5437c83a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:53,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:53,758 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,760 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:53,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:53,760 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:53,761 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,761 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:53,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:53,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289013762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:53,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289013762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:53,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289013763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:53,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289013763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:53,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289013764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,765 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:53,765 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:53,765 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,765 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:53,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:53,767 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:53,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:53,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:53,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:53,767 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:53,769 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,769 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:53,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,769 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:53,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,771 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:53,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,773 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:53,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:53,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:53,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:53,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:53,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:53,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:53,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:53,773 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:53,773 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:53,780 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/204d1c834d9a4f658d15e1329a46f5a0 2023-03-31 18:55:53,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:53,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289013864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:53,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289013864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:53,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289013864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:53,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289013865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:53,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289013865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,874 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:53,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:53,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:53,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:53,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:53,880 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:53,880 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:53,880 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:53,880 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:53,880 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,881 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:53,884 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,884 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:53,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:53,884 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,884 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:53,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:53,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:53,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:53,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:53,885 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:53,885 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:53,885 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:53,886 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,886 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:53,886 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:53,886 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:53,887 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,887 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:53,888 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,888 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:53,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:53,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c0b117[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:53,889 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:53,889 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,891 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:53,891 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:53,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:53,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:53,892 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:53,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:53,895 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:53,895 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:53,895 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:53,895 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:53,895 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:53,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:53,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:53,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,901 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,901 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:53,901 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:53,902 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:53,902 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,902 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:53,902 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:53,902 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:53,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:53,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:53,902 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:53,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:53,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:53,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:53,902 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:53,902 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:53,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:54,002 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:54,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:54,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:54,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:54,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:54,012 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:54,012 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:54,012 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:54,012 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:54,013 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,013 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,014 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:54,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:54,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:54,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:54,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:54,016 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:54,016 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:54,016 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:54,016 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:54,016 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:54,016 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:54,016 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:54,016 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:54,016 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:54,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:54,018 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cc1171[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:54,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:54,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,021 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:54,022 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:54,022 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:54,022 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:54,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:54,025 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:54,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:54,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:54,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:54,026 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:54,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:54,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:54,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,034 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:54,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:54,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:54,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:54,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:54,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:54,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:54,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,042 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:54,042 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:54,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:54,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289014066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289014069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289014070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289014071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289014071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,143 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:54,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:54,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:54,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:54,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:54,146 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:54,146 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:54,146 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:54,146 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:54,147 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,148 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,149 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:54,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:54,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:54,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:54,152 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,152 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:54,152 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:54,152 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:54,152 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:54,153 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:54,156 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:54,157 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,157 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@414ea4ed[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:54,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:54,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,161 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:54,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:54,168 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:54,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:54,172 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:54,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:54,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:54,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:54,172 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:54,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:54,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:54,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:54,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,183 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:54,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:54,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:54,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:54,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:54,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:54,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:54,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:54,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:54,186 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:54,186 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:54,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:54,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/be8723ddee5b47bd984e0952d9051f52 2023-03-31 18:55:54,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/97e004c07a9f4b55b7482bbc89a9fbfa 2023-03-31 18:55:54,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/204d1c834d9a4f658d15e1329a46f5a0 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/204d1c834d9a4f658d15e1329a46f5a0 2023-03-31 18:55:54,229 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/204d1c834d9a4f658d15e1329a46f5a0, entries=150, sequenceid=14, filesize=11.6 K 2023-03-31 18:55:54,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/be8723ddee5b47bd984e0952d9051f52 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/be8723ddee5b47bd984e0952d9051f52 2023-03-31 18:55:54,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/be8723ddee5b47bd984e0952d9051f52, entries=150, sequenceid=14, filesize=11.6 K 2023-03-31 18:55:54,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/97e004c07a9f4b55b7482bbc89a9fbfa as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/97e004c07a9f4b55b7482bbc89a9fbfa 2023-03-31 18:55:54,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/97e004c07a9f4b55b7482bbc89a9fbfa, entries=150, sequenceid=14, filesize=11.6 K 2023-03-31 18:55:54,238 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 91eb2b257640c6018ad8eeccf0deb2cf in 488ms, sequenceid=14, compaction requested=false 2023-03-31 18:55:54,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:54,286 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:54,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:54,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:54,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:54,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:54,290 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:54,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:54,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:54,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:54,291 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,291 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,292 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:54,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:54,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,293 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,293 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:54,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:54,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:54,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:54,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:54,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:54,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:54,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:54,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:54,295 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:54,295 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:54,295 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:55:54,296 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:55:54,296 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:54,296 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:55:54,296 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:54,296 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:55:54,296 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:54,306 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/97f40eadc1734a98b12eeb27fd0992d1 2023-03-31 18:55:54,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:54,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:54,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289014381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289014382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289014385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289014385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289014385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289014486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289014487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289014487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289014491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289014491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289014687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289014691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289014692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289014692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:54,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289014693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:54,734 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/fbb9332e519e4b9991b50da21a417492 2023-03-31 18:55:55,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289014994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289014996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289014997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289014997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289014999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,144 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/791bbf757f524087aa22e0f99b5371d8 2023-03-31 18:55:55,148 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/97f40eadc1734a98b12eeb27fd0992d1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/97f40eadc1734a98b12eeb27fd0992d1 2023-03-31 18:55:55,151 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/97f40eadc1734a98b12eeb27fd0992d1, entries=150, sequenceid=34, filesize=11.6 K 2023-03-31 18:55:55,151 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/fbb9332e519e4b9991b50da21a417492 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/fbb9332e519e4b9991b50da21a417492 2023-03-31 18:55:55,154 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/fbb9332e519e4b9991b50da21a417492, entries=150, sequenceid=34, filesize=11.6 K 2023-03-31 18:55:55,155 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/791bbf757f524087aa22e0f99b5371d8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/791bbf757f524087aa22e0f99b5371d8 2023-03-31 18:55:55,158 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/791bbf757f524087aa22e0f99b5371d8, entries=150, sequenceid=34, filesize=11.6 K 2023-03-31 18:55:55,158 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 91eb2b257640c6018ad8eeccf0deb2cf in 863ms, sequenceid=34, compaction requested=false 2023-03-31 18:55:55,158 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:55,158 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:55,160 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:55,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd2c2e6[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:55,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:55,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:55,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:55,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:55,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:55,165 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:55,168 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:55,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:55,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:55,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:55,168 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:55,170 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:55,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:55,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,181 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:55,181 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:55,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:55,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,281 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:55,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:55,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:55,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:55,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:55,284 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:55,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:55,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:55,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,286 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,286 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:55,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:55,287 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,287 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:55,287 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 18:55:55,288 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:55:55,288 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:55,288 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:55:55,288 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:55,288 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:55:55,288 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:55,294 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/b060a41d60984f12a4a7238eccc6f52d 2023-03-31 18:55:55,304 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/ff688b39096f42b29272c21d07f5f191 2023-03-31 18:55:55,312 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/18b71d45c85d4e228b67f742abf7fafd 2023-03-31 18:55:55,315 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/b060a41d60984f12a4a7238eccc6f52d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b060a41d60984f12a4a7238eccc6f52d 2023-03-31 18:55:55,318 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b060a41d60984f12a4a7238eccc6f52d, entries=150, sequenceid=47, filesize=11.6 K 2023-03-31 18:55:55,318 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/ff688b39096f42b29272c21d07f5f191 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ff688b39096f42b29272c21d07f5f191 2023-03-31 18:55:55,323 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ff688b39096f42b29272c21d07f5f191, entries=150, sequenceid=47, filesize=11.6 K 2023-03-31 18:55:55,324 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/18b71d45c85d4e228b67f742abf7fafd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/18b71d45c85d4e228b67f742abf7fafd 2023-03-31 18:55:55,327 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/18b71d45c85d4e228b67f742abf7fafd, entries=150, sequenceid=47, filesize=11.6 K 2023-03-31 18:55:55,328 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 91eb2b257640c6018ad8eeccf0deb2cf in 41ms, sequenceid=47, compaction requested=true 2023-03-31 18:55:55,328 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:55,328 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:55,333 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:55,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fde9682[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:55,334 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:55,334 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,337 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:55,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:55,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:55,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:55,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:55,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:55,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:55,341 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:55,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:55,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:55,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:55,341 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:55,344 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,344 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,344 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,345 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,349 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:55,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:55,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,350 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:55,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,350 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:55,350 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,350 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,450 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:55,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:55,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:55,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:55,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:55,453 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:55,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:55,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:55,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,458 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,458 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:55,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:55,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:55,460 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,460 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:55,460 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:55,460 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,461 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:55,461 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:55,461 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,461 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:55,461 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:55,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,462 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,462 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,462 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a3c5591[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:55,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:55,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,465 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:55,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:55,466 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:55,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,466 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,466 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:55,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:55,471 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:55,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:55,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:55,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:55,471 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:55,476 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,476 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,478 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,480 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:55,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:55,484 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,484 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,484 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,484 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:55,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:55,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:55:55,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:55:55,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:55,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:55:55,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:55,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:55:55,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:55,534 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/78ac82fc77564700b84196ecbd6d10db 2023-03-31 18:55:55,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289015540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680288955517, max timestamp=1680288955541Num uniques 859; 2023-03-31 18:55:55,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 18:55:55,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680288955517, max timestamp=1680288955541Num uniques 859; 2023-03-31 18:55:55,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680288955517, max timestamp=1680288955541Num uniques 859; 2023-03-31 18:55:55,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 18:55:55,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 18:55:55,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/a7f00c5cb02243128aff795b1f127978 2023-03-31 18:55:55,584 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:55,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:55,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:55,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:55,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:55,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289015578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,588 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:55,588 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,588 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:55,588 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:55,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289015578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,588 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,588 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,589 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,589 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,590 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,590 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:55,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:55,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:55,591 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:55,591 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,591 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:55,591 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:55,591 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:55,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289015579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,591 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,592 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:55,592 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:55,592 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:55,593 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,593 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,594 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:55,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e61f9f7[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:55,595 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:55,595 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289015588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:55,598 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,599 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:55,600 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:55,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:55,603 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:55,603 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:55,603 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:55,603 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:55,603 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:55,604 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,604 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,604 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,605 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,608 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:55,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:55,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:55,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,610 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,611 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:55,611 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:55,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289015644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289015689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289015689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289015692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289015698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,711 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:55,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:55,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:55,714 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:55,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:55,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:55,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,717 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,717 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,717 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,717 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:55,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:55,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:55,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:55,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:55,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:55,719 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,719 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:55,719 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:55,719 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:55,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:55,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:55,719 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:55,720 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,720 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,721 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,721 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:55,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4399c12f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:55,722 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:55,722 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,723 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,723 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:55,723 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:55,723 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:55,724 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,724 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:55,726 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:55,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:55,729 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:55,729 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:55,729 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:55,729 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:55,729 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:55,732 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,732 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,732 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,733 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,735 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,737 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,737 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,737 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,737 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,737 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,737 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,738 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:55,737 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:55,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:55,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,738 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:55,738 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:55,838 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:55,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:55,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:55,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:55,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:55,841 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:55,841 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,841 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:55,841 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:55,842 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,842 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,843 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,843 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,843 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,843 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:55,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:55,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:55,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:55,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:55,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:55,845 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,845 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:55,845 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:55,845 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:55,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:55,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:55,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:55,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e82558[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:55,852 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:55,853 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,854 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:55,854 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,854 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:55,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289015851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:55,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,855 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,855 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:55,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:55,858 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:55,858 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:55,858 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:55,858 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:55,858 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:55,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,859 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,860 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,864 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,868 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,868 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,868 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,868 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:55,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:55,868 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,868 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,868 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,868 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,868 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,868 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,868 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:55,868 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:55,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:55,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289015898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289015899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289015905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:55,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289015910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,969 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:55,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:55,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:55,971 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:55,972 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,972 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:55,972 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:55,972 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,972 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,973 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,973 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,973 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,973 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:55,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:55,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:55,975 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,975 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:55,975 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:55,975 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:55,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/d0014c81eb9943deaf5d2433b997b92b 2023-03-31 18:55:55,977 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:55,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d06309[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:55,978 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:55,979 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,980 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:55,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:55,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:55,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/78ac82fc77564700b84196ecbd6d10db as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78ac82fc77564700b84196ecbd6d10db 2023-03-31 18:55:55,981 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,981 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:55,982 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,982 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:55,982 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,982 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78ac82fc77564700b84196ecbd6d10db, entries=200, sequenceid=59, filesize=13.9 K 2023-03-31 18:55:55,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/a7f00c5cb02243128aff795b1f127978 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a7f00c5cb02243128aff795b1f127978 2023-03-31 18:55:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:55,985 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:55,985 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:55,985 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:55,985 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:55,985 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:55,986 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,986 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,986 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:55,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:55,987 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:55,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a7f00c5cb02243128aff795b1f127978, entries=150, sequenceid=59, filesize=11.6 K 2023-03-31 18:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/d0014c81eb9943deaf5d2433b997b92b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d0014c81eb9943deaf5d2433b997b92b 2023-03-31 18:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,991 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d0014c81eb9943deaf5d2433b997b92b, entries=150, sequenceid=59, filesize=11.6 K 2023-03-31 18:55:55,991 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 91eb2b257640c6018ad8eeccf0deb2cf in 480ms, sequenceid=59, compaction requested=true 2023-03-31 18:55:55,991 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,992 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:55,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:55,992 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:55,992 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:55,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:55,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:55,993 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49884 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:55,993 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:55,993 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:55:55,993 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:55:55,993 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,993 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:55,993 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/204d1c834d9a4f658d15e1329a46f5a0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/97f40eadc1734a98b12eeb27fd0992d1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b060a41d60984f12a4a7238eccc6f52d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78ac82fc77564700b84196ecbd6d10db] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=48.7 K 2023-03-31 18:55:55,993 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/be8723ddee5b47bd984e0952d9051f52, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/fbb9332e519e4b9991b50da21a417492, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ff688b39096f42b29272c21d07f5f191, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a7f00c5cb02243128aff795b1f127978] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=46.4 K 2023-03-31 18:55:55,993 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 204d1c834d9a4f658d15e1329a46f5a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680288953746 2023-03-31 18:55:55,993 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting be8723ddee5b47bd984e0952d9051f52, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680288953746 2023-03-31 18:55:55,994 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 97f40eadc1734a98b12eeb27fd0992d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680288953761 2023-03-31 18:55:55,994 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting fbb9332e519e4b9991b50da21a417492, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680288953761 2023-03-31 18:55:55,994 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting b060a41d60984f12a4a7238eccc6f52d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680288954380 2023-03-31 18:55:55,994 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ff688b39096f42b29272c21d07f5f191, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680288954380 2023-03-31 18:55:55,994 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 78ac82fc77564700b84196ecbd6d10db, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680288955509 2023-03-31 18:55:55,994 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a7f00c5cb02243128aff795b1f127978, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680288955509 2023-03-31 18:55:55,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:55,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:55,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:55,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:55,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:55,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:55,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:55,999 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:55,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:56,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:56,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:56,000 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:56,000 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:56,002 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#387 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:56,003 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#388 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:56,009 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/fdc0f42da5904a5a9cf9635421016a10 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/fdc0f42da5904a5a9cf9635421016a10 2023-03-31 18:55:56,012 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into fdc0f42da5904a5a9cf9635421016a10(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:56,012 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:56,012 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=12, startTime=1680288955992; duration=0sec 2023-03-31 18:55:56,012 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:56,012 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:56,013 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:56,013 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:55:56,013 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:56,013 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/97e004c07a9f4b55b7482bbc89a9fbfa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/791bbf757f524087aa22e0f99b5371d8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/18b71d45c85d4e228b67f742abf7fafd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d0014c81eb9943deaf5d2433b997b92b] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=46.4 K 2023-03-31 18:55:56,013 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 97e004c07a9f4b55b7482bbc89a9fbfa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680288953746 2023-03-31 18:55:56,013 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 791bbf757f524087aa22e0f99b5371d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680288953761 2023-03-31 18:55:56,014 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 18b71d45c85d4e228b67f742abf7fafd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680288954380 2023-03-31 18:55:56,014 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d0014c81eb9943deaf5d2433b997b92b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680288955509 2023-03-31 18:55:56,019 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#389 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:56,100 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:56,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:56,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:56,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:56,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:56,103 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:56,103 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:56,103 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:56,103 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:56,103 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:56,103 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:56,104 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,104 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:56,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:56,105 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,105 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:56,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:56,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:56,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:56,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:56,106 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:56,106 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:56,106 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:56,106 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:56,106 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:56,106 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:56,106 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:56,106 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-03-31 18:55:56,107 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:55:56,107 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:56,107 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:55:56,107 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:56,107 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:55:56,107 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:56,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:56,171 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:56,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289016209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289016209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289016211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289016212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289016216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289016313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289016316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289016320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,412 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/d2c406938768440faf55dc73e3d5b975 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/d2c406938768440faf55dc73e3d5b975 2023-03-31 18:55:56,416 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into d2c406938768440faf55dc73e3d5b975(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:56,416 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:56,416 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=12, startTime=1680288955992; duration=0sec 2023-03-31 18:55:56,417 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:56,426 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/b793c6d321494322b4373035069f6ac1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/b793c6d321494322b4373035069f6ac1 2023-03-31 18:55:56,432 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into b793c6d321494322b4373035069f6ac1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:56,432 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:56,432 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=12, startTime=1680288955992; duration=0sec 2023-03-31 18:55:56,432 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:56,514 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/54e25367321a458aa345bb8f9f972e14 2023-03-31 18:55:56,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289016521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289016523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289016523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289016720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289016727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289016828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289016828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:56,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289016829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,947 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/3caf35ee3fe944e19d7dd4fa352d97f6 2023-03-31 18:55:56,964 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/53df9b15bee346de8e074120585b7ad7 2023-03-31 18:55:56,968 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/54e25367321a458aa345bb8f9f972e14 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/54e25367321a458aa345bb8f9f972e14 2023-03-31 18:55:56,971 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/54e25367321a458aa345bb8f9f972e14, entries=150, sequenceid=94, filesize=11.6 K 2023-03-31 18:55:56,971 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/3caf35ee3fe944e19d7dd4fa352d97f6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3caf35ee3fe944e19d7dd4fa352d97f6 2023-03-31 18:55:56,976 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3caf35ee3fe944e19d7dd4fa352d97f6, entries=150, sequenceid=94, filesize=11.6 K 2023-03-31 18:55:56,981 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/53df9b15bee346de8e074120585b7ad7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/53df9b15bee346de8e074120585b7ad7 2023-03-31 18:55:56,984 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/53df9b15bee346de8e074120585b7ad7, entries=150, sequenceid=94, filesize=11.6 K 2023-03-31 18:55:56,985 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 91eb2b257640c6018ad8eeccf0deb2cf in 879ms, sequenceid=94, compaction requested=false 2023-03-31 18:55:56,985 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:56,985 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:56,985 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:56,985 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:56,985 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:56,990 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:56,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:56,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f92a5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:56,991 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:56,991 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:56,992 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:56,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:56,993 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,993 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:56,994 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:56,997 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:56,997 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:56,997 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:56,997 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:56,997 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:56,998 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:56,998 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:56,998 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:56,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:56,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:56,999 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:57,000 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:57,003 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:57,003 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:57,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:57,003 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,003 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:57,003 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:57,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:57,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:57,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:57,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:57,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:57,004 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:57,004 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:57,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:57,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:57,004 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:57,104 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:57,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:57,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:57,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:57,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:57,107 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:57,108 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:57,108 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:57,108 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:57,108 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:57,108 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:57,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:57,110 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:57,110 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,110 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:57,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:57,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:57,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:57,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:57,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:57,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:57,111 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:57,111 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:57,111 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:57,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:57,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:57,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288957360, max timestamp=1680288957377Num uniques 859; 2023-03-31 18:55:57,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:57,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288957360, max timestamp=1680288957377Num uniques 859; 2023-03-31 18:55:57,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:57,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288957360, max timestamp=1680288957377Num uniques 859; 2023-03-31 18:55:57,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:55:57,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289017416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289017417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289017418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,522 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/8927416edac04274b13e0e48ea00245d 2023-03-31 18:55:57,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289017530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289017530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289017532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,574 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/5a2985b499414c33a0d47ff4ade3d047 2023-03-31 18:55:57,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289017730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289017732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289017732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289017740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:57,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:57,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289017744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,028 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/31c10f2381ee47f9bf680b9ad9a66789 2023-03-31 18:55:58,034 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/8927416edac04274b13e0e48ea00245d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/8927416edac04274b13e0e48ea00245d 2023-03-31 18:55:58,039 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/8927416edac04274b13e0e48ea00245d, entries=150, sequenceid=107, filesize=11.6 K 2023-03-31 18:55:58,040 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/5a2985b499414c33a0d47ff4ade3d047 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/5a2985b499414c33a0d47ff4ade3d047 2023-03-31 18:55:58,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289018042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,051 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/5a2985b499414c33a0d47ff4ade3d047, entries=150, sequenceid=107, filesize=11.6 K 2023-03-31 18:55:58,052 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/31c10f2381ee47f9bf680b9ad9a66789 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/31c10f2381ee47f9bf680b9ad9a66789 2023-03-31 18:55:58,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289018044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,055 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/31c10f2381ee47f9bf680b9ad9a66789, entries=150, sequenceid=107, filesize=11.6 K 2023-03-31 18:55:58,056 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 91eb2b257640c6018ad8eeccf0deb2cf in 945ms, sequenceid=107, compaction requested=true 2023-03-31 18:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:58,058 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:58,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c12ab12[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:58,060 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:58,060 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:58,062 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-31 18:55:58,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:55:58,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:58,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:55:58,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:58,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:55:58,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:58,062 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:58,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:58,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:58,063 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,063 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:58,071 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:58,073 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:58,073 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:58,073 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:58,073 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:58,073 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:58,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:58,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,077 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,079 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:58,079 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:58,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:58,080 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,080 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,080 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,080 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,080 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,080 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:58,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:58,080 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:58,080 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,080 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:58,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:58,080 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:58,099 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/eda9b9c36833480aa4585d37450dee46 2023-03-31 18:55:58,129 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/81ee9051b2b24aff814055c0d0b7371c 2023-03-31 18:55:58,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289018151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/a671ec556fc4433f9ad5335ae7b3c980 2023-03-31 18:55:58,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/eda9b9c36833480aa4585d37450dee46 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/eda9b9c36833480aa4585d37450dee46 2023-03-31 18:55:58,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/eda9b9c36833480aa4585d37450dee46, entries=200, sequenceid=142, filesize=14.1 K 2023-03-31 18:55:58,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/81ee9051b2b24aff814055c0d0b7371c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/81ee9051b2b24aff814055c0d0b7371c 2023-03-31 18:55:58,180 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:58,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:58,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:58,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:58,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:58,190 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:58,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:58,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:58,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:58,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,191 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:58,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:58,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:58,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:58,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:58,205 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:58,205 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:58,205 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:58,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/81ee9051b2b24aff814055c0d0b7371c, entries=150, sequenceid=142, filesize=11.8 K 2023-03-31 18:55:58,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:58,206 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:58,206 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,206 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:58,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/a671ec556fc4433f9ad5335ae7b3c980 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a671ec556fc4433f9ad5335ae7b3c980 2023-03-31 18:55:58,206 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:58,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a671ec556fc4433f9ad5335ae7b3c980, entries=150, sequenceid=142, filesize=11.8 K 2023-03-31 18:55:58,210 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:58,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:58,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,210 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 91eb2b257640c6018ad8eeccf0deb2cf in 148ms, sequenceid=142, compaction requested=true 2023-03-31 18:55:58,210 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:58,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:58,211 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:58,211 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:58,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:58,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:58,212 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:58,212 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:55:58,212 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:58,212 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:55:58,212 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,212 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,212 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/fdc0f42da5904a5a9cf9635421016a10, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/54e25367321a458aa345bb8f9f972e14, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/8927416edac04274b13e0e48ea00245d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/eda9b9c36833480aa4585d37450dee46] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=49.0 K 2023-03-31 18:55:58,212 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/d2c406938768440faf55dc73e3d5b975, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3caf35ee3fe944e19d7dd4fa352d97f6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/5a2985b499414c33a0d47ff4ade3d047, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/81ee9051b2b24aff814055c0d0b7371c] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=46.7 K 2023-03-31 18:55:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:58,213 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting fdc0f42da5904a5a9cf9635421016a10, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680288955509 2023-03-31 18:55:58,214 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 54e25367321a458aa345bb8f9f972e14, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680288955563 2023-03-31 18:55:58,214 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d2c406938768440faf55dc73e3d5b975, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680288955509 2023-03-31 18:55:58,214 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8927416edac04274b13e0e48ea00245d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680288956207 2023-03-31 18:55:58,214 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,215 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting eda9b9c36833480aa4585d37450dee46, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680288957402 2023-03-31 18:55:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,215 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c98d2d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:58,217 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:58,217 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:58,215 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3caf35ee3fe944e19d7dd4fa352d97f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680288955563 2023-03-31 18:55:58,218 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:58,219 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,219 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:58,219 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 5a2985b499414c33a0d47ff4ade3d047, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680288956207 2023-03-31 18:55:58,220 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 81ee9051b2b24aff814055c0d0b7371c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680288957402 2023-03-31 18:55:58,220 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:58,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:58,223 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:58,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:58,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:58,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:58,223 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:58,225 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,225 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:58,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,225 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,227 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,230 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:58,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:58,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:58,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,237 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:58,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:58,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:58,238 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:58,238 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:58,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,238 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,238 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:58,242 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#399 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:58,248 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 18:55:58,253 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#400 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:58,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:58,279 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:55:58,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:55:58,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:58,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:55:58,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:58,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:55:58,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:58,307 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/c79a36079ab943c0a20657fe301886fc as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/c79a36079ab943c0a20657fe301886fc 2023-03-31 18:55:58,311 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into c79a36079ab943c0a20657fe301886fc(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:58,311 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:58,311 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=12, startTime=1680288958211; duration=0sec 2023-03-31 18:55:58,311 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:55:58,311 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:55:58,313 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:55:58,313 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:55:58,313 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,313 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/b793c6d321494322b4373035069f6ac1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/53df9b15bee346de8e074120585b7ad7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/31c10f2381ee47f9bf680b9ad9a66789, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a671ec556fc4433f9ad5335ae7b3c980] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=46.7 K 2023-03-31 18:55:58,313 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b793c6d321494322b4373035069f6ac1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680288955509 2023-03-31 18:55:58,314 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 53df9b15bee346de8e074120585b7ad7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680288955563 2023-03-31 18:55:58,314 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 31c10f2381ee47f9bf680b9ad9a66789, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680288956207 2023-03-31 18:55:58,314 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a671ec556fc4433f9ad5335ae7b3c980, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680288957402 2023-03-31 18:55:58,338 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:58,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:58,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:58,358 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#402 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:55:58,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:58,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:58,360 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:58,360 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:58,360 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:58,360 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:58,361 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,361 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,362 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:58,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,363 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:58,364 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,364 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:58,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:58,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:58,392 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:58,392 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:58,392 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,392 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:58,392 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:58,392 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,392 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:58,392 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:58,392 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:58,394 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,394 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,395 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,395 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,398 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/4ed04bd63553465b9f784d920706c788 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4ed04bd63553465b9f784d920706c788 2023-03-31 18:55:58,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:58,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66726cb6[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:58,398 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:58,398 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,400 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,400 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:58,401 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:58,403 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into 4ed04bd63553465b9f784d920706c788(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:58,403 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:58,403 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=12, startTime=1680288958211; duration=0sec 2023-03-31 18:55:58,403 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:58,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:58,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:58,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:58,404 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:58,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:58,407 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:58,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:58,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:58,407 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:58,407 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:58,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:58,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,412 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,414 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:58,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:58,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:58,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:58,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:58,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:58,415 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:58,415 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:58,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:58,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:58,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:58,416 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288958411, max timestamp=1680288958467Num uniques 859; 2023-03-31 18:55:58,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288958411, max timestamp=1680288958467Num uniques 859; 2023-03-31 18:55:58,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288958411, max timestamp=1680288958467Num uniques 859; 2023-03-31 18:55:58,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:55:58,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:55:58,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:55:58,515 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:58,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:58,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:58,519 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:58,519 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:58,519 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:58,519 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:58,519 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,520 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,522 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,522 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:58,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:58,522 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,522 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:58,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:58,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:58,524 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:58,524 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:58,524 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:58,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:58,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:58,534 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,534 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:58,535 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:58,535 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:58,538 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,539 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,539 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:58,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b154067[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:58,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:58,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,541 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:58,543 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:58,543 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:58,543 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:58,543 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,544 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:58,544 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:58,544 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:58,544 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:58,544 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:58,544 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:58,544 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:58,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:58,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,549 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,550 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,552 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:58,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:58,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:58,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:58,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:58,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:58,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:58,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,556 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,556 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,556 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:58,556 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:58,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289018581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289018585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289018591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,656 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:58,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:58,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:58,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:58,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:58,664 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:58,664 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:58,664 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:58,664 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:58,665 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,665 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,666 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,666 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:58,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:58,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:58,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:58,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:58,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:58,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:58,681 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:58,683 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c6222c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:58,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:58,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,688 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,689 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:58,689 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:58,689 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:58,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:58,690 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:58,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:58,695 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:58,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:58,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:58,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:58,695 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:58,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:58,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:58,698 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289018692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289018698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289018699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:58,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:58,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:58,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,705 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:58,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:58,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:58,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:58,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:58,706 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:58,706 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:58,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:55:58,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/43074f0c5e6a4b80a820f2e2678111aa 2023-03-31 18:55:58,735 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/c08fcabd6af74ba48c96a10d62e6e202 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c08fcabd6af74ba48c96a10d62e6e202 2023-03-31 18:55:58,744 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into c08fcabd6af74ba48c96a10d62e6e202(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:55:58,744 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:58,744 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=12, startTime=1680288958211; duration=0sec 2023-03-31 18:55:58,744 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:55:58,767 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/6146e039e3e84efda8f3b5b813222b71 2023-03-31 18:55:58,792 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/77ee22d973474b5cb0cee4fa5489f35e 2023-03-31 18:55:58,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/43074f0c5e6a4b80a820f2e2678111aa as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43074f0c5e6a4b80a820f2e2678111aa 2023-03-31 18:55:58,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43074f0c5e6a4b80a820f2e2678111aa, entries=200, sequenceid=153, filesize=14.1 K 2023-03-31 18:55:58,806 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:58,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:58,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/6146e039e3e84efda8f3b5b813222b71 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6146e039e3e84efda8f3b5b813222b71 2023-03-31 18:55:58,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:58,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:58,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:58,810 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:58,810 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:58,810 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:58,810 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:58,810 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,810 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6146e039e3e84efda8f3b5b813222b71, entries=150, sequenceid=153, filesize=11.8 K 2023-03-31 18:55:58,815 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/77ee22d973474b5cb0cee4fa5489f35e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/77ee22d973474b5cb0cee4fa5489f35e 2023-03-31 18:55:58,817 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:58,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:58,817 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,817 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,818 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:58,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:58,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:58,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/77ee22d973474b5cb0cee4fa5489f35e, entries=150, sequenceid=153, filesize=11.8 K 2023-03-31 18:55:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:58,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 91eb2b257640c6018ad8eeccf0deb2cf in 545ms, sequenceid=153, compaction requested=false 2023-03-31 18:55:58,824 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:58,824 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:58,824 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:58,824 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-31 18:55:58,824 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:55:58,824 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:58,824 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:55:58,825 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:58,825 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:55:58,825 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:55:58,872 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/42c605b5b2c04e8cacab6ed05e5edbeb 2023-03-31 18:55:58,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:58,918 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:58,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289018951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289018954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:58,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:58,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289018955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289019064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289019066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289019072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289019274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289019275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289019278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,332 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/edf2adaca8484ea2b8bfbac92fb94316 2023-03-31 18:55:59,349 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/9731184289a545deb1041e0a45b8eb74 2023-03-31 18:55:59,359 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/42c605b5b2c04e8cacab6ed05e5edbeb as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/42c605b5b2c04e8cacab6ed05e5edbeb 2023-03-31 18:55:59,363 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/42c605b5b2c04e8cacab6ed05e5edbeb, entries=150, sequenceid=192, filesize=11.8 K 2023-03-31 18:55:59,364 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/edf2adaca8484ea2b8bfbac92fb94316 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/edf2adaca8484ea2b8bfbac92fb94316 2023-03-31 18:55:59,368 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/edf2adaca8484ea2b8bfbac92fb94316, entries=150, sequenceid=192, filesize=11.8 K 2023-03-31 18:55:59,369 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/9731184289a545deb1041e0a45b8eb74 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9731184289a545deb1041e0a45b8eb74 2023-03-31 18:55:59,377 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9731184289a545deb1041e0a45b8eb74, entries=150, sequenceid=192, filesize=11.8 K 2023-03-31 18:55:59,378 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 91eb2b257640c6018ad8eeccf0deb2cf in 554ms, sequenceid=192, compaction requested=true 2023-03-31 18:55:59,378 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:55:59,378 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:55:59,383 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:59,383 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:55:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:59,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:55:59,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b3f67bc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:55:59,386 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:55:59,386 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:59,387 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:59,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:55:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:55:59,387 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,387 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:55:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:55:59,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:55:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:55:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:55:59,392 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:55:59,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:55:59,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:55:59,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:55:59,392 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:55:59,395 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:55:59,397 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:59,397 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:59,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:59,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:55:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:55:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:55:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:59,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:59,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,409 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:55:59,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:55:59,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:55:59,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:59,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:59,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:55:59,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:59,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:59,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:55:59,409 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:55:59,409 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:55:59,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:55:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:55:59,510 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:55:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:55:59,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:55:59,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:55:59,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:55:59,513 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:55:59,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:59,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:55:59,513 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:55:59,521 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:59,521 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:59,523 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:55:59,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:55:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:55:59,524 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:59,525 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,525 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:55:59,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:55:59,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:55:59,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:55:59,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:55:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:55:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:55:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:55:59,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:55:59,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:55:59,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:55:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:55:59,530 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:55:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:55:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:55:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:55:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:55:59,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:55:59,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:55:59,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289019648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289019650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288959630, max timestamp=1680288959659Num uniques 859; 2023-03-31 18:55:59,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:55:59,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288959630, max timestamp=1680288959659Num uniques 859; 2023-03-31 18:55:59,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:55:59,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288959630, max timestamp=1680288959659Num uniques 859; 2023-03-31 18:55:59,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:55:59,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289019763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289019773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289019774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289019775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33136 deadline: 1680289019775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,792 DEBUG [Thread-3003] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4202 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:55:59,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289019876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289019890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289019891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:55:59,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289019891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:55:59,955 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/638325e7ebf84ed392d998501960464f 2023-03-31 18:55:59,971 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/6469673853d948b58af430776267c5e1 2023-03-31 18:55:59,999 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/e5672c8632694429977552eb95dea18c 2023-03-31 18:56:00,004 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/638325e7ebf84ed392d998501960464f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/638325e7ebf84ed392d998501960464f 2023-03-31 18:56:00,008 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/638325e7ebf84ed392d998501960464f, entries=150, sequenceid=203, filesize=11.8 K 2023-03-31 18:56:00,009 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/6469673853d948b58af430776267c5e1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6469673853d948b58af430776267c5e1 2023-03-31 18:56:00,015 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6469673853d948b58af430776267c5e1, entries=150, sequenceid=203, filesize=11.8 K 2023-03-31 18:56:00,016 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/e5672c8632694429977552eb95dea18c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e5672c8632694429977552eb95dea18c 2023-03-31 18:56:00,019 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e5672c8632694429977552eb95dea18c, entries=150, sequenceid=203, filesize=11.8 K 2023-03-31 18:56:00,019 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 91eb2b257640c6018ad8eeccf0deb2cf in 489ms, sequenceid=203, compaction requested=true 2023-03-31 18:56:00,019 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:00,020 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:00,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:00,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:00,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:00,021 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2edda7b1[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:00,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:00,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,031 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:00,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:00,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:00,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,032 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:00,033 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,033 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:00,037 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:00,037 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:00,037 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:00,037 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:00,037 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:00,039 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,039 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,039 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,040 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:00,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:00,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,048 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,048 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:00,048 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:00,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:00,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-03-31 18:56:00,100 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:00,100 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:00,100 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:00,100 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:00,100 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:00,100 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:00,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/24f28baaa40a45a79a15a3eb889c0430 2023-03-31 18:56:00,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289020114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289020120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289020121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289020128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,148 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:00,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:00,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:00,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:00,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:00,151 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:00,151 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,151 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:00,151 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:00,152 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,152 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,153 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,153 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,153 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,153 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:00,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:00,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:00,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:00,159 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:00,159 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:00,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:00,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:00,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:00,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ffadc75[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:00,164 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:00,164 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,165 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:00,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:00,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,166 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:00,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:00,168 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:00,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:00,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:00,170 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:00,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:00,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:00,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:00,170 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:00,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:00,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:00,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,177 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:00,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,177 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:00,178 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289020228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289020230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289020236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289020238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,278 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:00,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:00,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:00,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:00,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:00,281 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:00,281 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,281 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:00,281 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:00,281 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,283 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,283 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:00,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:00,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:00,285 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,285 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:00,285 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:00,285 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,287 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1e5ae4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:00,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:00,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,291 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:00,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:00,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:00,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:00,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:00,298 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:00,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:00,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:00,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:00,298 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:00,299 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,299 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,299 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,301 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,301 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,303 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,304 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:00,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:00,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,304 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:00,304 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:00,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:00,404 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:00,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:00,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:00,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:00,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:00,410 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:00,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:00,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:00,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,413 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:00,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:00,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:00,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:00,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:00,429 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,429 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,429 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:00,429 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cbcca5c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:00,431 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:00,431 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,432 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:00,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:00,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:00,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:00,434 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:00,440 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:00,440 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:00,440 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:00,440 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:00,441 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:00,444 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,444 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,444 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289020437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289020437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,449 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289020443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289020446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,462 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:00,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:00,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,462 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:00,462 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:00,540 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/818b93c267984bda86676f85db42f53b 2023-03-31 18:56:00,557 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/3ca152cb754445ec8feaee90b9c771be 2023-03-31 18:56:00,562 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:00,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/24f28baaa40a45a79a15a3eb889c0430 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/24f28baaa40a45a79a15a3eb889c0430 2023-03-31 18:56:00,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:00,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:00,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:00,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:00,566 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:00,566 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,566 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:00,566 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:00,568 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,568 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:00,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:00,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:00,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/24f28baaa40a45a79a15a3eb889c0430, entries=200, sequenceid=239, filesize=14.1 K 2023-03-31 18:56:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:00,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/818b93c267984bda86676f85db42f53b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/818b93c267984bda86676f85db42f53b 2023-03-31 18:56:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:00,574 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,574 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:00,574 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:00,574 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:00,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:00,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:00,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:00,579 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f1f0455[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:00,583 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:00,583 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,584 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:00,585 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,585 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:00,586 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/818b93c267984bda86676f85db42f53b, entries=150, sequenceid=239, filesize=11.8 K 2023-03-31 18:56:00,587 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/3ca152cb754445ec8feaee90b9c771be as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3ca152cb754445ec8feaee90b9c771be 2023-03-31 18:56:00,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:00,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:00,590 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:00,590 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:00,590 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:00,590 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:00,590 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:00,593 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,593 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,593 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,594 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3ca152cb754445ec8feaee90b9c771be, entries=150, sequenceid=239, filesize=11.8 K 2023-03-31 18:56:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,595 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,596 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=46.96 KB/48090 for 91eb2b257640c6018ad8eeccf0deb2cf in 496ms, sequenceid=239, compaction requested=true 2023-03-31 18:56:00,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:00,596 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:00,596 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:56:00,596 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:56:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,596 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:00,597 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,598 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60454 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:56:00,598 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:00,598 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,598 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/c79a36079ab943c0a20657fe301886fc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6146e039e3e84efda8f3b5b813222b71, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/edf2adaca8484ea2b8bfbac92fb94316, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6469673853d948b58af430776267c5e1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/818b93c267984bda86676f85db42f53b] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=59.0 K 2023-03-31 18:56:00,598 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 65234 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:56:00,599 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:00,599 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,599 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c08fcabd6af74ba48c96a10d62e6e202, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43074f0c5e6a4b80a820f2e2678111aa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/42c605b5b2c04e8cacab6ed05e5edbeb, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/638325e7ebf84ed392d998501960464f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/24f28baaa40a45a79a15a3eb889c0430] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=63.7 K 2023-03-31 18:56:00,601 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting c79a36079ab943c0a20657fe301886fc, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680288957402 2023-03-31 18:56:00,601 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c08fcabd6af74ba48c96a10d62e6e202, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680288957402 2023-03-31 18:56:00,601 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 6146e039e3e84efda8f3b5b813222b71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680288958125 2023-03-31 18:56:00,603 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting edf2adaca8484ea2b8bfbac92fb94316, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680288958579 2023-03-31 18:56:00,603 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 43074f0c5e6a4b80a820f2e2678111aa, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680288958125 2023-03-31 18:56:00,603 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 6469673853d948b58af430776267c5e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680288958948 2023-03-31 18:56:00,604 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 818b93c267984bda86676f85db42f53b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680288959721 2023-03-31 18:56:00,604 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 42c605b5b2c04e8cacab6ed05e5edbeb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680288958579 2023-03-31 18:56:00,605 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 638325e7ebf84ed392d998501960464f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680288958948 2023-03-31 18:56:00,605 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 24f28baaa40a45a79a15a3eb889c0430, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680288959721 2023-03-31 18:56:00,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,608 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,609 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,609 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,609 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:00,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:00,610 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:00,610 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:00,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:00,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,618 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#414 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:00,633 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#415 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:00,655 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/b0839aeac64249b6ab47be8994472604 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b0839aeac64249b6ab47be8994472604 2023-03-31 18:56:00,662 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into b0839aeac64249b6ab47be8994472604(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:00,662 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:00,662 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=11, startTime=1680288960596; duration=0sec 2023-03-31 18:56:00,662 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:00,662 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:56:00,664 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60454 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:56:00,665 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:00,665 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,665 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4ed04bd63553465b9f784d920706c788, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/77ee22d973474b5cb0cee4fa5489f35e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9731184289a545deb1041e0a45b8eb74, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e5672c8632694429977552eb95dea18c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3ca152cb754445ec8feaee90b9c771be] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=59.0 K 2023-03-31 18:56:00,665 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 4ed04bd63553465b9f784d920706c788, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680288957402 2023-03-31 18:56:00,666 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 77ee22d973474b5cb0cee4fa5489f35e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680288958125 2023-03-31 18:56:00,666 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 9731184289a545deb1041e0a45b8eb74, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680288958579 2023-03-31 18:56:00,667 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting e5672c8632694429977552eb95dea18c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680288958948 2023-03-31 18:56:00,667 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 3ca152cb754445ec8feaee90b9c771be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680288959721 2023-03-31 18:56:00,677 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#416 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:00,689 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/bf65d00c94724257ace347e611a156c1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/bf65d00c94724257ace347e611a156c1 2023-03-31 18:56:00,696 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into bf65d00c94724257ace347e611a156c1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:00,696 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:00,696 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=11, startTime=1680288960597; duration=0sec 2023-03-31 18:56:00,696 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:00,710 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:00,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:00,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:00,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:00,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:00,715 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:00,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:00,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:00,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,718 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,718 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,719 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,719 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:00,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:00,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:00,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:00,730 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,730 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:00,730 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:56:00,730 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:00,730 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:00,730 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:00,730 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:00,731 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:00,731 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:00,750 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:00,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:00,782 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/7aa9d512aee346188c5d835a9f60b96e 2023-03-31 18:56:00,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288960763, max timestamp=1680288960803Num uniques 859; 2023-03-31 18:56:00,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288960791, max timestamp=1680288960803Num uniques 859; 2023-03-31 18:56:00,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:00,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:56:00,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288960791, max timestamp=1680288960803Num uniques 859; 2023-03-31 18:56:00,808 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/094e4149252841c9ad563996e58e7985 2023-03-31 18:56:00,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:00,833 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/61b381d7a3a54a94a170a33797974bec 2023-03-31 18:56:00,838 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/7aa9d512aee346188c5d835a9f60b96e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/7aa9d512aee346188c5d835a9f60b96e 2023-03-31 18:56:00,842 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/7aa9d512aee346188c5d835a9f60b96e, entries=150, sequenceid=251, filesize=11.8 K 2023-03-31 18:56:00,843 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/094e4149252841c9ad563996e58e7985 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/094e4149252841c9ad563996e58e7985 2023-03-31 18:56:00,847 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/094e4149252841c9ad563996e58e7985, entries=150, sequenceid=251, filesize=11.8 K 2023-03-31 18:56:00,847 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/61b381d7a3a54a94a170a33797974bec as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/61b381d7a3a54a94a170a33797974bec 2023-03-31 18:56:00,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289020849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,860 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/61b381d7a3a54a94a170a33797974bec, entries=150, sequenceid=251, filesize=11.8 K 2023-03-31 18:56:00,861 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 91eb2b257640c6018ad8eeccf0deb2cf in 131ms, sequenceid=251, compaction requested=false 2023-03-31 18:56:00,861 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:00,861 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:00,864 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,865 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=143.13 KB heapSize=405.30 KB 2023-03-31 18:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:00,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64f60b7[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:00,866 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:00,866 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:00,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 18:56:00,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:00,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 18:56:00,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:00,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 18:56:00,869 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,869 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:00,869 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:00,869 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:00,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,874 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:00,874 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:00,874 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:00,874 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:00,874 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:00,874 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:00,874 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:00,878 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,878 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,878 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,880 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,881 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,885 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,885 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,885 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,885 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,885 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:00,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:00,885 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:00,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,885 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:00,885 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,886 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:00,886 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,886 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,886 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:00,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:00,903 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/ddee0ab6ca5d4613b4d7d8fdc436c5e5 2023-03-31 18:56:00,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289020892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289020892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289020893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:00,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289020953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,986 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:00,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:00,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:00,989 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:00,989 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,989 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:00,989 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:00,990 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,990 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,991 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:00,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:00,991 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,992 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,992 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:00,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:00,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:00,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:00,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:00,995 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,995 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:00,995 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:00,995 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:00,996 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,996 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:00,997 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,997 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:00,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:00,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d8d3e0[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:00,998 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:00,998 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,999 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:00,999 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:00,999 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:00,999 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:00,999 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:00,999 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:01,002 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:01,002 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:01,002 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:01,002 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:01,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:01,004 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:01,004 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:01,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:01,005 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:01,005 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:01,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:01,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,006 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:01,007 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,008 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,010 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:01,010 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:01,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:01,011 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:01,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:01,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:01,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:01,011 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:01,011 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:01,011 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,011 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,011 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,011 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,011 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,011 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,011 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:01,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289021008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289021008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289021008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,053 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/02ac0bbce212478590d862c5994b330c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/02ac0bbce212478590d862c5994b330c 2023-03-31 18:56:01,057 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into 02ac0bbce212478590d862c5994b330c(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:01,057 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:01,057 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=11, startTime=1680288960596; duration=0sec 2023-03-31 18:56:01,057 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:01,111 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:01,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:01,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:01,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:01,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:01,115 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:01,115 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:01,115 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:01,115 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:01,115 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,115 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,116 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,116 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:01,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:01,117 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,117 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:01,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:01,118 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:01,118 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:01,118 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:01,118 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:01,118 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:01,118 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:01,119 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:01,119 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:01,119 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:01,119 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:01,119 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:01,119 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,119 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:01,119 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:01,125 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,125 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:01,125 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:01,125 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:01,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:01,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32dc1c8d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:01,130 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:01,130 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,133 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,134 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:01,134 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:01,134 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:01,134 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,134 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:01,135 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:01,135 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:01,135 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:01,136 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:01,138 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:01,139 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:01,139 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:01,139 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:01,139 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:01,140 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,140 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:01,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,141 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:01,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,144 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:01,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,150 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,154 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:01,154 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:01,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:01,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:01,155 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,155 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,156 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:01,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:01,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:01,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,156 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:01,156 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:01,156 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289021159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33150 deadline: 1680289021218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289021218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33156 deadline: 1680289021219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,256 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:01,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:01,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:01,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:01,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:01,259 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:01,260 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:01,260 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:01,260 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:01,265 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,266 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,267 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:01,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,268 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,268 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,268 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:01,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:01,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:01,270 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:01,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:01,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:01,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:01,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:01,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:01,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,273 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:01,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:01,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:01,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5569b2f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:01,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:01,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,277 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,277 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:01,277 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:01,277 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:01,278 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,278 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:01,280 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,280 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:01,280 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:01,280 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:01,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:01,282 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:01,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:01,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:01,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:01,282 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:01,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:01,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,284 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:01,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:01,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:01,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:01,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:01,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:01,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:01,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:01,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:01,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,290 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:01,290 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:01,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:01,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:01,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:01,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:01,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/2f97de4b7de24d61908ebf79146e1f6c 2023-03-31 18:56:01,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/c36100c76af94f62a8ab691a3ec45e1d 2023-03-31 18:56:01,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/ddee0ab6ca5d4613b4d7d8fdc436c5e5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ddee0ab6ca5d4613b4d7d8fdc436c5e5 2023-03-31 18:56:01,353 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ddee0ab6ca5d4613b4d7d8fdc436c5e5, entries=150, sequenceid=289, filesize=11.9 K 2023-03-31 18:56:01,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/2f97de4b7de24d61908ebf79146e1f6c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2f97de4b7de24d61908ebf79146e1f6c 2023-03-31 18:56:01,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2f97de4b7de24d61908ebf79146e1f6c, entries=150, sequenceid=289, filesize=11.9 K 2023-03-31 18:56:01,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/c36100c76af94f62a8ab691a3ec45e1d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/c36100c76af94f62a8ab691a3ec45e1d 2023-03-31 18:56:01,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/c36100c76af94f62a8ab691a3ec45e1d, entries=150, sequenceid=289, filesize=11.9 K 2023-03-31 18:56:01,366 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~425.88 KB/436096, currentSize=33.54 KB/34350 for 91eb2b257640c6018ad8eeccf0deb2cf in 501ms, sequenceid=289, compaction requested=true 2023-03-31 18:56:01,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:01,366 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:01,366 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:01,366 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:01,366 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:01,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:01,367 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:01,367 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:01,367 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:01,367 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:01,367 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:01,367 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:01,367 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b0839aeac64249b6ab47be8994472604, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/7aa9d512aee346188c5d835a9f60b96e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ddee0ab6ca5d4613b4d7d8fdc436c5e5] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=35.8 K 2023-03-31 18:56:01,367 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/02ac0bbce212478590d862c5994b330c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/094e4149252841c9ad563996e58e7985, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2f97de4b7de24d61908ebf79146e1f6c] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=35.8 K 2023-03-31 18:56:01,368 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting b0839aeac64249b6ab47be8994472604, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680288959721 2023-03-31 18:56:01,368 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 02ac0bbce212478590d862c5994b330c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680288959721 2023-03-31 18:56:01,368 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 7aa9d512aee346188c5d835a9f60b96e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680288960111 2023-03-31 18:56:01,368 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 094e4149252841c9ad563996e58e7985, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680288960111 2023-03-31 18:56:01,368 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting ddee0ab6ca5d4613b4d7d8fdc436c5e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680288960847 2023-03-31 18:56:01,368 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2f97de4b7de24d61908ebf79146e1f6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680288960847 2023-03-31 18:56:01,375 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:01,377 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#424 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:01,390 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:01,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:01,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:01,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:01,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:01,400 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:01,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:01,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:01,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:01,402 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,402 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,404 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:01,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:01,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:01,404 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:01,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:01,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:01,409 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/40709ac3a3b64458b4a57e345d9b97c9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40709ac3a3b64458b4a57e345d9b97c9 2023-03-31 18:56:01,414 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into 40709ac3a3b64458b4a57e345d9b97c9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:01,414 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:01,414 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=13, startTime=1680288961366; duration=0sec 2023-03-31 18:56:01,414 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:01,414 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:01,415 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:01,415 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:01,415 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:01,416 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/bf65d00c94724257ace347e611a156c1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/61b381d7a3a54a94a170a33797974bec, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/c36100c76af94f62a8ab691a3ec45e1d] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=35.8 K 2023-03-31 18:56:01,416 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting bf65d00c94724257ace347e611a156c1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680288959721 2023-03-31 18:56:01,416 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 61b381d7a3a54a94a170a33797974bec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680288960111 2023-03-31 18:56:01,417 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c36100c76af94f62a8ab691a3ec45e1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680288960847 2023-03-31 18:56:01,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:01,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:01,429 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:01,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:01,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:01,432 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:01,432 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:01,432 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-31 18:56:01,432 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:01,433 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:01,433 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:01,433 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:01,433 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:01,433 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:01,440 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/03e21819bc1d4e5199076983ca884804 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03e21819bc1d4e5199076983ca884804 2023-03-31 18:56:01,444 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into 03e21819bc1d4e5199076983ca884804(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:01,444 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:01,445 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=13, startTime=1680288961366; duration=0sec 2023-03-31 18:56:01,445 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:01,470 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/43bb42247d764e28a8831443263f0747 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/43bb42247d764e28a8831443263f0747 2023-03-31 18:56:01,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:01,472 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:01,475 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into 43bb42247d764e28a8831443263f0747(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:01,475 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:01,475 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=13, startTime=1680288961367; duration=0sec 2023-03-31 18:56:01,475 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:01,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680288961482, max timestamp=1680288961545Num uniques 859; 2023-03-31 18:56:01,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:01,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680288961482, max timestamp=1680288961545Num uniques 859; 2023-03-31 18:56:01,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:01,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288961482, max timestamp=1680288961545Num uniques 859; 2023-03-31 18:56:01,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:01,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288961573, max timestamp=1680288961615Num uniques 1068; 2023-03-31 18:56:01,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-31 18:56:01,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288961573, max timestamp=1680288961615Num uniques 1068; 2023-03-31 18:56:01,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-31 18:56:01,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680288961573, max timestamp=1680288961615Num uniques 1068; 2023-03-31 18:56:01,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-31 18:56:01,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680288961630, max timestamp=1680288961668Num uniques 1027; 2023-03-31 18:56:01,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-31 18:56:01,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680288961630, max timestamp=1680288961668Num uniques 1027; 2023-03-31 18:56:01,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-31 18:56:01,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680288961630, max timestamp=1680288961668Num uniques 1027; 2023-03-31 18:56:01,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-31 18:56:01,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289021707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289021708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289021714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289021716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289021817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289021818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289021822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:01,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289021822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:01,864 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/a5952a20c13f4deeb147103bfe24178a 2023-03-31 18:56:01,873 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/3695dd0d155f4aa5b523b3d8b7e8d681 2023-03-31 18:56:02,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289022026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289022026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289022027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289022027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,289 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/cb153303bbd949019c1c757bffa61a17 2023-03-31 18:56:02,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/a5952a20c13f4deeb147103bfe24178a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/a5952a20c13f4deeb147103bfe24178a 2023-03-31 18:56:02,297 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/a5952a20c13f4deeb147103bfe24178a, entries=150, sequenceid=299, filesize=11.9 K 2023-03-31 18:56:02,298 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/3695dd0d155f4aa5b523b3d8b7e8d681 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3695dd0d155f4aa5b523b3d8b7e8d681 2023-03-31 18:56:02,302 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3695dd0d155f4aa5b523b3d8b7e8d681, entries=150, sequenceid=299, filesize=11.9 K 2023-03-31 18:56:02,303 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/cb153303bbd949019c1c757bffa61a17 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/cb153303bbd949019c1c757bffa61a17 2023-03-31 18:56:02,307 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/cb153303bbd949019c1c757bffa61a17, entries=150, sequenceid=299, filesize=11.9 K 2023-03-31 18:56:02,307 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 91eb2b257640c6018ad8eeccf0deb2cf in 875ms, sequenceid=299, compaction requested=false 2023-03-31 18:56:02,307 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:02,307 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,307 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:02,307 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:02,307 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:02,308 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:02,311 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,311 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,311 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,311 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:02,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786109ea[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:02,313 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:02,313 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,315 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:02,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:02,315 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:02,315 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,315 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:02,316 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,316 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:02,316 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:02,316 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:02,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:02,318 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:02,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:02,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:02,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:02,318 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:02,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:02,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,322 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,323 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,326 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:02,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:02,326 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:02,326 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:02,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:02,326 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:02,327 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:02,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,327 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:02,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:02,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:02,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680288961705, max timestamp=1680288962336Num uniques 1136; 2023-03-31 18:56:02,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-31 18:56:02,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:02,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680288961705, max timestamp=1680288962336Num uniques 1136; 2023-03-31 18:56:02,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-31 18:56:02,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=76.04 KB heapSize=207.19 KB 2023-03-31 18:56:02,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680288961705, max timestamp=1680288962336Num uniques 1136; 2023-03-31 18:56:02,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-31 18:56:02,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:02,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:02,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:02,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:02,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:02,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:02,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/d9c31b315bea4a8bbc1f0ad09875c06b 2023-03-31 18:56:02,372 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/f7f55a4b50184b46a422d6da77e6d269 2023-03-31 18:56:02,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288962379, max timestamp=1680288962390Num uniques 859; 2023-03-31 18:56:02,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288962379, max timestamp=1680288962390Num uniques 859; 2023-03-31 18:56:02,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:02,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288962379, max timestamp=1680288962390Num uniques 859; 2023-03-31 18:56:02,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:02,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:02,427 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:02,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:02,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289022424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:02,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:02,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:02,430 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:02,431 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:02,431 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:02,431 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:02,431 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,431 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,432 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:02,432 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:02,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:02,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:02,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:02,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:02,435 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,435 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:02,435 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:02,435 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:02,437 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:02,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b892d5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:02,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:02,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289022427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,441 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:02,441 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:02,441 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:02,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,441 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:02,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289022428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289022428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,443 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:02,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:02,446 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:02,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:02,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:02,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:02,446 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:02,447 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,447 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,447 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:02,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:02,448 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,450 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:02,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:02,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:02,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:02,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:02,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:02,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,454 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:02,454 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:02,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:02,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289022529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289022542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289022543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289022543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,554 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:02,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:02,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:02,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:02,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:02,559 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:02,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:02,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:02,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:02,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:02,562 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,562 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,562 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:02,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:02,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:02,572 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,572 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:02,572 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:02,572 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:02,574 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34d2c2fb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:02,576 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:02,576 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,577 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:02,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:02,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:02,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:02,586 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:02,586 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:02,586 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:02,586 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:02,586 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:02,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,589 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:02,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:02,589 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,592 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,595 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:02,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:02,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:02,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:02,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:02,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:02,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:02,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,595 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:02,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,595 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:02,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:02,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:02,696 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:02,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:02,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:02,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:02,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:02,699 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:02,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:02,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:02,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:02,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:02,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:02,701 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,701 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,701 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:02,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:02,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:02,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:02,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:02,702 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:02,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:02,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:02,703 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,703 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:02,703 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:02,703 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:02,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:02,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:02,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:02,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:02,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ecb8ad[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:02,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:02,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,709 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:02,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:02,711 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:02,711 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,711 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:02,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:02,715 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:02,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:02,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:02,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:02,715 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:02,717 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,717 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:02,717 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:02,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:02,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:02,719 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,724 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,726 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:02,726 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:02,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:02,727 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:02,727 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,727 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,727 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,727 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,727 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:02,727 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:02,727 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:02,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:02,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:02,728 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:02,728 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:02,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289022734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289022749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289022749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:02,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289022750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,782 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/fdc183b8ba5e48e6af92edf77861baf3 2023-03-31 18:56:02,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/d9c31b315bea4a8bbc1f0ad09875c06b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d9c31b315bea4a8bbc1f0ad09875c06b 2023-03-31 18:56:02,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d9c31b315bea4a8bbc1f0ad09875c06b, entries=250, sequenceid=376, filesize=16.7 K 2023-03-31 18:56:02,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/f7f55a4b50184b46a422d6da77e6d269 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f7f55a4b50184b46a422d6da77e6d269 2023-03-31 18:56:02,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f7f55a4b50184b46a422d6da77e6d269, entries=150, sequenceid=376, filesize=11.9 K 2023-03-31 18:56:02,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/fdc183b8ba5e48e6af92edf77861baf3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/fdc183b8ba5e48e6af92edf77861baf3 2023-03-31 18:56:02,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/fdc183b8ba5e48e6af92edf77861baf3, entries=150, sequenceid=376, filesize=11.9 K 2023-03-31 18:56:02,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~123.42 KB/126384, currentSize=140.89 KB/144270 for 91eb2b257640c6018ad8eeccf0deb2cf in 460ms, sequenceid=376, compaction requested=true 2023-03-31 18:56:02,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:02,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:02,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:02,799 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:02,799 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:02,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:02,800 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41984 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:02,801 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:02,801 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:02,801 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:02,801 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,801 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,801 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40709ac3a3b64458b4a57e345d9b97c9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/a5952a20c13f4deeb147103bfe24178a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d9c31b315bea4a8bbc1f0ad09875c06b] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=41 K 2023-03-31 18:56:02,801 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03e21819bc1d4e5199076983ca884804, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3695dd0d155f4aa5b523b3d8b7e8d681, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f7f55a4b50184b46a422d6da77e6d269] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.2 K 2023-03-31 18:56:02,801 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 40709ac3a3b64458b4a57e345d9b97c9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680288960847 2023-03-31 18:56:02,801 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 03e21819bc1d4e5199076983ca884804, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680288960847 2023-03-31 18:56:02,802 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a5952a20c13f4deeb147103bfe24178a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680288960877 2023-03-31 18:56:02,802 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d9c31b315bea4a8bbc1f0ad09875c06b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680288961705 2023-03-31 18:56:02,802 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3695dd0d155f4aa5b523b3d8b7e8d681, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680288960877 2023-03-31 18:56:02,804 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f7f55a4b50184b46a422d6da77e6d269, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680288961706 2023-03-31 18:56:02,811 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#432 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:02,812 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#433 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:02,829 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:02,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:02,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:02,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:02,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:02,848 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:02,848 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:02,848 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:02,848 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:02,849 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,849 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,851 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:02,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:02,852 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:02,852 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:02,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:02,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:02,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:02,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:02,854 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-31 18:56:02,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:02,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:02,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:02,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:02,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:02,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:02,854 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/4262c4a6c47b4803ba39f90a61f76681 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4262c4a6c47b4803ba39f90a61f76681 2023-03-31 18:56:02,862 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into 4262c4a6c47b4803ba39f90a61f76681(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:02,862 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:02,862 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=13, startTime=1680288962799; duration=0sec 2023-03-31 18:56:02,863 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:02,863 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:02,863 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:02,863 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:02,864 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:02,864 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/43bb42247d764e28a8831443263f0747, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/cb153303bbd949019c1c757bffa61a17, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/fdc183b8ba5e48e6af92edf77861baf3] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.2 K 2023-03-31 18:56:02,864 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 43bb42247d764e28a8831443263f0747, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680288960847 2023-03-31 18:56:02,864 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting cb153303bbd949019c1c757bffa61a17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680288960877 2023-03-31 18:56:02,865 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting fdc183b8ba5e48e6af92edf77861baf3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680288961706 2023-03-31 18:56:02,905 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#435 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:02,908 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/c1e9954d47bb4a098d8458f4d8bf83f2 2023-03-31 18:56:02,928 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/0085128a12e54914a84445d4b9b715ae as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/0085128a12e54914a84445d4b9b715ae 2023-03-31 18:56:02,935 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into 0085128a12e54914a84445d4b9b715ae(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:02,935 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:02,935 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=13, startTime=1680288962799; duration=0sec 2023-03-31 18:56:02,935 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:02,937 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/ed08b6cde848413daa487d4351961d55 2023-03-31 18:56:03,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:03,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:03,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289023096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289023097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289023108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289023108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33188 deadline: 1680289023209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289023209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289023218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289023219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,230 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/685143671c14496db0e3994ba5d79cf7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/685143671c14496db0e3994ba5d79cf7 2023-03-31 18:56:03,235 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into 685143671c14496db0e3994ba5d79cf7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:03,235 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:03,235 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=13, startTime=1680288962799; duration=0sec 2023-03-31 18:56:03,235 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:03,354 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/6efd9765e7184d938693031ebc22d077 2023-03-31 18:56:03,358 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/c1e9954d47bb4a098d8458f4d8bf83f2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c1e9954d47bb4a098d8458f4d8bf83f2 2023-03-31 18:56:03,362 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c1e9954d47bb4a098d8458f4d8bf83f2, entries=150, sequenceid=413, filesize=11.9 K 2023-03-31 18:56:03,363 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/ed08b6cde848413daa487d4351961d55 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed08b6cde848413daa487d4351961d55 2023-03-31 18:56:03,367 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed08b6cde848413daa487d4351961d55, entries=150, sequenceid=413, filesize=11.9 K 2023-03-31 18:56:03,368 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/6efd9765e7184d938693031ebc22d077 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/6efd9765e7184d938693031ebc22d077 2023-03-31 18:56:03,371 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/6efd9765e7184d938693031ebc22d077, entries=150, sequenceid=413, filesize=11.9 K 2023-03-31 18:56:03,372 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 91eb2b257640c6018ad8eeccf0deb2cf in 518ms, sequenceid=413, compaction requested=false 2023-03-31 18:56:03,372 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:03,372 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:03,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:03,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:03,373 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:03,374 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,374 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:03,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75b2cfa3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:03,376 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:03,376 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,377 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:03,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:03,378 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:03,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,380 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:03,380 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:03,380 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:03,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:03,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:03,380 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:03,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:03,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:03,380 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:03,380 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:03,386 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,386 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:03,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,386 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,388 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,389 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,391 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:03,391 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:03,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:03,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:03,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,392 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,392 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:03,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:03,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:03,393 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:03,393 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:03,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:03,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:03,418 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:56:03,418 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:03,418 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:03,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:03,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:03,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:03,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:03,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/47ee6c1ca319478c8bc49863a6bf2577 2023-03-31 18:56:03,465 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/46ef460961b944899130f9831c99b816 2023-03-31 18:56:03,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289023459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288963457, max timestamp=1680288963468Num uniques 859; 2023-03-31 18:56:03,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:03,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288963457, max timestamp=1680288963468Num uniques 859; 2023-03-31 18:56:03,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:03,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288963457, max timestamp=1680288963468Num uniques 859; 2023-03-31 18:56:03,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:03,486 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/736855a50e184ffb8a251da5cf82b370 2023-03-31 18:56:03,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/47ee6c1ca319478c8bc49863a6bf2577 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/47ee6c1ca319478c8bc49863a6bf2577 2023-03-31 18:56:03,493 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:03,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:03,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:03,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:03,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:03,498 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:03,498 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:03,498 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:03,498 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:03,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,500 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/47ee6c1ca319478c8bc49863a6bf2577, entries=200, sequenceid=428, filesize=14.3 K 2023-03-31 18:56:03,500 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:03,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:03,500 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,500 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/46ef460961b944899130f9831c99b816 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/46ef460961b944899130f9831c99b816 2023-03-31 18:56:03,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:03,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:03,502 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:03,502 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:03,502 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:03,504 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,504 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:03,504 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:03,504 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:03,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/46ef460961b944899130f9831c99b816, entries=150, sequenceid=428, filesize=11.9 K 2023-03-31 18:56:03,507 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/736855a50e184ffb8a251da5cf82b370 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/736855a50e184ffb8a251da5cf82b370 2023-03-31 18:56:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,509 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,509 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57bed5fa[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:03,510 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:03,510 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:03,512 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,512 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:03,513 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:03,513 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:03,513 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:03,513 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/736855a50e184ffb8a251da5cf82b370, entries=150, sequenceid=428, filesize=11.9 K 2023-03-31 18:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,515 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=93.93 KB/96180 for 91eb2b257640c6018ad8eeccf0deb2cf in 97ms, sequenceid=428, compaction requested=true 2023-03-31 18:56:03,515 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:03,515 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:03,515 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:03,515 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,515 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:03,516 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:03,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:03,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=100.63 KB heapSize=285.38 KB 2023-03-31 18:56:03,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:03,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:03,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:03,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:03,517 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,518 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:03,518 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:03,518 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,518 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4262c4a6c47b4803ba39f90a61f76681, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed08b6cde848413daa487d4351961d55, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/46ef460961b944899130f9831c99b816] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.3 K 2023-03-31 18:56:03,519 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:03,519 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:03,519 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,519 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/685143671c14496db0e3994ba5d79cf7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c1e9954d47bb4a098d8458f4d8bf83f2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/47ee6c1ca319478c8bc49863a6bf2577] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=38.7 K 2023-03-31 18:56:03,519 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4262c4a6c47b4803ba39f90a61f76681, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680288961706 2023-03-31 18:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:03,519 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 685143671c14496db0e3994ba5d79cf7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680288961706 2023-03-31 18:56:03,519 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:03,520 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:03,520 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:03,520 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:03,520 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ed08b6cde848413daa487d4351961d55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680288962412 2023-03-31 18:56:03,520 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c1e9954d47bb4a098d8458f4d8bf83f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680288962412 2023-03-31 18:56:03,520 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:03,520 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 46ef460961b944899130f9831c99b816, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680288963106 2023-03-31 18:56:03,520 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 47ee6c1ca319478c8bc49863a6bf2577, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680288963069 2023-03-31 18:56:03,523 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,524 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:03,524 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,528 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,529 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:03,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:03,533 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:03,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:03,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:03,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,533 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,533 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:03,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:03,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:03,533 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:03,535 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:03,539 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#443 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:03,565 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/403fd6a8570848debc1d3997b4334b17 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/403fd6a8570848debc1d3997b4334b17 2023-03-31 18:56:03,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289023559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289023559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,570 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into 403fd6a8570848debc1d3997b4334b17(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:03,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289023560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,570 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:03,570 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=13, startTime=1680288963515; duration=0sec 2023-03-31 18:56:03,570 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:03,570 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:03,572 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:03,572 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:03,572 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,573 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/0085128a12e54914a84445d4b9b715ae, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/6efd9765e7184d938693031ebc22d077, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/736855a50e184ffb8a251da5cf82b370] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.3 K 2023-03-31 18:56:03,573 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 0085128a12e54914a84445d4b9b715ae, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680288961706 2023-03-31 18:56:03,573 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 6efd9765e7184d938693031ebc22d077, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680288962412 2023-03-31 18:56:03,574 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 736855a50e184ffb8a251da5cf82b370, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680288963106 2023-03-31 18:56:03,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289023572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,583 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#444 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:03,597 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/636a672236a3438abf015cea5d6380ce as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/636a672236a3438abf015cea5d6380ce 2023-03-31 18:56:03,603 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into 636a672236a3438abf015cea5d6380ce(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:03,603 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:03,603 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=13, startTime=1680288963516; duration=0sec 2023-03-31 18:56:03,603 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:03,635 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:03,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:03,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:03,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:03,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:03,638 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:03,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:03,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:03,638 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:03,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,639 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,640 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:03,640 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:03,640 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,640 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:03,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:03,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:03,642 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:03,642 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:03,642 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:03,642 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:03,643 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,643 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:03,643 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:03,643 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:03,644 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,645 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,645 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:03,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755900ec[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:03,646 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:03,646 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,647 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:03,647 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,647 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:03,651 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:03,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,651 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:03,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,651 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:03,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:03,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:03,653 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:03,653 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:03,653 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:03,653 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:03,653 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:03,655 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,655 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,655 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:03,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,655 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,658 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,659 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,659 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,659 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,659 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:03,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:03,660 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:03,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:03,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:03,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,660 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:03,660 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:03,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:03,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:03,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:03,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:03,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289023669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289023670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289023672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,760 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:03,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:03,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:03,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:03,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:03,764 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:03,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:03,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:03,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:03,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,764 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,766 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:03,766 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:03,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:03,766 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,766 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:03,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:03,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:03,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:03,767 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:03,768 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,768 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:03,768 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:03,768 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:03,771 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,771 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,772 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:03,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1510f73a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:03,773 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:03,773 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:03,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:03,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:03,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:03,776 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,776 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:03,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,776 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:03,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:03,779 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:03,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:03,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:03,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:03,779 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:03,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,783 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:03,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,785 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289023782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:03,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:03,793 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:03,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:03,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:03,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:03,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:03,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,794 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:03,794 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:03,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:03,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33112 deadline: 1680289023821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,830 DEBUG [Thread-3003] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8240 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:56:03,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289023876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289023876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:03,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289023882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,894 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:03,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:03,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:03,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:03,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:03,897 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:03,897 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:03,897 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:03,897 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:03,898 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,898 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:03,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:03,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:03,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:03,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:03,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:03,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:03,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:03,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:03,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:03,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:03,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:03,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:03,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:03,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:03,902 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:03,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:03,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:03,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:03,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:03,903 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,904 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:03,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:03,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763cedec[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:03,905 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:03,905 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,907 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,907 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:03,907 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:03,907 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:03,907 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:03,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:03,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:03,909 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:03,909 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:03,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:03,913 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:03,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:03,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:03,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:03,913 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:03,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,914 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:03,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:03,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:03,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:03,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:03,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:03,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:03,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:03,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:03,920 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:03,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:03,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:03,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:03,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:03,921 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:03,921 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:03,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/5acb6d4c7a6c48d2a03e5e992919481d 2023-03-31 18:56:03,960 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/7fc3e3d7d8fe4445908ecefc8c2e856c 2023-03-31 18:56:03,964 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/2705966f8f8e4a2987a876e393eb64a5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2705966f8f8e4a2987a876e393eb64a5 2023-03-31 18:56:03,972 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into 2705966f8f8e4a2987a876e393eb64a5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:03,972 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:03,972 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=13, startTime=1680288963515; duration=0sec 2023-03-31 18:56:03,972 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:04,021 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:04,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:04,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:04,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:04,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:04,026 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:04,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:04,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:04,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:04,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,028 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:04,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:04,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:04,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:04,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:04,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:04,030 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:04,030 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:04,031 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:04,031 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:04,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:04,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:04,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:04,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:04,032 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,034 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,034 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:04,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@578ce7d4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:04,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:04,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,037 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:04,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:04,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:04,037 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,037 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:04,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:04,039 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:04,044 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:04,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:04,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:04,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:04,044 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:04,047 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,047 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:04,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,047 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:04,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,049 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:04,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:04,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:04,052 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:04,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:04,052 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:04,052 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:04,052 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:04,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:04,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:04,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:04,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289024089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,152 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:04,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:04,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:04,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:04,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:04,156 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:04,156 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:04,156 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:04,156 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:04,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,158 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:04,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:04,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:04,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:04,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:04,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:04,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:04,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:04,160 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:04,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:04,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:04,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6fd3a9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:04,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:04,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,167 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:04,167 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,167 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:04,169 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:04,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:04,171 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:04,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:04,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:04,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:04,171 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:04,173 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,174 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,174 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:04,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,176 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,183 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:04,183 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:04,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:04,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:04,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:04,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:04,183 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:04,183 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:04,183 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:04,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:04,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289024186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:04,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289024187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:04,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289024188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,283 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:04,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:04,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:04,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:04,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:04,287 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:04,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:04,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:04,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:04,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:04,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:04,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:04,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,289 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:04,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:04,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:04,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:04,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:04,290 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:04,290 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:04,290 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:04,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:04,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:04,292 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:04,292 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:04,292 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:04,292 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:04,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:04,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:04,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:04,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:04,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,294 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:04,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785440f9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:04,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:04,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:04,299 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:04,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:04,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:04,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:04,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:04,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:04,304 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:04,304 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:04,305 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:04,305 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:04,305 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:04,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:04,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:04,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,308 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:04,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:04,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:04,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,311 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,311 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:04,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:04,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:04,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:04,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:04,311 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:04,311 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:04,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/ecc0d9fefffd4a24b0a5f4de3b12bb32 2023-03-31 18:56:04,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/5acb6d4c7a6c48d2a03e5e992919481d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5acb6d4c7a6c48d2a03e5e992919481d 2023-03-31 18:56:04,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5acb6d4c7a6c48d2a03e5e992919481d, entries=200, sequenceid=460, filesize=14.3 K 2023-03-31 18:56:04,384 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/7fc3e3d7d8fe4445908ecefc8c2e856c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/7fc3e3d7d8fe4445908ecefc8c2e856c 2023-03-31 18:56:04,388 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/7fc3e3d7d8fe4445908ecefc8c2e856c, entries=150, sequenceid=460, filesize=11.9 K 2023-03-31 18:56:04,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/ecc0d9fefffd4a24b0a5f4de3b12bb32 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/ecc0d9fefffd4a24b0a5f4de3b12bb32 2023-03-31 18:56:04,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/ecc0d9fefffd4a24b0a5f4de3b12bb32, entries=150, sequenceid=460, filesize=11.9 K 2023-03-31 18:56:04,393 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~285.33 KB/292176, currentSize=80.51 KB/82440 for 91eb2b257640c6018ad8eeccf0deb2cf in 877ms, sequenceid=460, compaction requested=false 2023-03-31 18:56:04,393 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:04,411 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:04,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:04,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:04,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:04,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:04,415 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:04,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:04,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:04,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:04,417 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,417 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,423 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:04,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:04,423 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,423 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:04,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:04,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:04,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:04,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:04,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:04,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:04,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:04,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:04,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:04,425 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:04,425 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:04,425 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-31 18:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:04,444 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/bea9dda17bd54f798dd7db20f84d24aa 2023-03-31 18:56:04,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:04,603 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:04,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:04,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289024684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:04,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289024692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:04,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289024695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:04,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289024699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:04,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289024795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,867 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/a59ff037f4444573a027b13a5c95f086 2023-03-31 18:56:04,876 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/15a1772f87dd4c19a5b616cd7df9b453 2023-03-31 18:56:04,880 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/bea9dda17bd54f798dd7db20f84d24aa as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/bea9dda17bd54f798dd7db20f84d24aa 2023-03-31 18:56:04,883 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/bea9dda17bd54f798dd7db20f84d24aa, entries=150, sequenceid=478, filesize=11.9 K 2023-03-31 18:56:04,884 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/a59ff037f4444573a027b13a5c95f086 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a59ff037f4444573a027b13a5c95f086 2023-03-31 18:56:04,888 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a59ff037f4444573a027b13a5c95f086, entries=150, sequenceid=478, filesize=11.9 K 2023-03-31 18:56:04,889 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/15a1772f87dd4c19a5b616cd7df9b453 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/15a1772f87dd4c19a5b616cd7df9b453 2023-03-31 18:56:04,893 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/15a1772f87dd4c19a5b616cd7df9b453, entries=150, sequenceid=478, filesize=11.9 K 2023-03-31 18:56:04,894 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 91eb2b257640c6018ad8eeccf0deb2cf in 469ms, sequenceid=478, compaction requested=true 2023-03-31 18:56:04,894 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:04,894 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:04,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:04,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:04,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69169192[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:04,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:04,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,903 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:04,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:04,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:04,905 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:04,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:04,906 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:04,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:04,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:04,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:04,906 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:04,910 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,910 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:04,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,910 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:04,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:04,911 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:04,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:04,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:04,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:04,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:04,917 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:04,917 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:04,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:04,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:04,916 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:04,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:04,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:04,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:04,917 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:04,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:05,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-31 18:56:05,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:05,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:05,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:05,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:05,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:05,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:05,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/c372dafd1ef44c229e477a89d06db22a 2023-03-31 18:56:05,017 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:05,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:05,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:05,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:05,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:05,022 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:05,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:05,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:05,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,023 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,025 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,026 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:05,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:05,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:05,027 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:05,027 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,027 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:05,028 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,028 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:05,028 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:05,028 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/abae065691e348b6ad4bd1e5dc2e3850 2023-03-31 18:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:05,031 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,031 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:05,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59946923[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:05,034 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:05,034 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:05,037 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:05,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:05,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:05,039 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:05,041 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:05,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:05,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:05,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:05,042 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:05,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,044 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,045 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:05,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,051 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:05,051 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:05,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:05,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289025083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,151 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:05,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:05,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:05,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:05,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:05,155 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:05,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:05,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:05,155 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,156 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,158 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:05,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:05,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:05,161 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,161 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:05,161 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:05,161 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:05,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:05,166 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:05,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df20b23[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:05,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:05,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,169 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:05,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:05,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:05,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:05,171 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:05,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,171 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,171 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:05,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:05,174 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:05,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:05,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:05,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:05,175 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:05,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,177 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,178 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,184 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:05,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:05,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:05,184 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:05,184 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:05,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,185 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289025195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,284 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:05,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:05,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:05,292 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:05,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:05,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:05,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,294 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,294 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,294 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:05,295 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:05,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:05,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:05,300 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:05,300 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,300 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:05,300 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:05,300 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:05,300 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,300 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:05,300 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:05,301 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:05,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,303 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,304 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@177bc9ec[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:05,304 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:05,305 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,306 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:05,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:05,309 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:05,309 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,309 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:05,310 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:05,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:05,314 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:05,314 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:05,314 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:05,314 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:05,314 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:05,316 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,316 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,316 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:05,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:05,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,323 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:05,323 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:05,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,322 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289025400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,423 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:05,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:05,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:05,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:05,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:05,427 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:05,427 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,427 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:05,427 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:05,427 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,427 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,429 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,429 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,430 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,430 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:05,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:05,431 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,431 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:05,431 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:05,431 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:05,434 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a3c5090[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:05,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:05,436 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,437 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:05,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:05,437 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:05,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:05,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,438 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,438 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:05,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:05,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:05,439 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:05,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:05,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:05,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:05,439 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:05,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/3d8fdecc991a4e13ab41914989f263c8 2023-03-31 18:56:05,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,447 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/c372dafd1ef44c229e477a89d06db22a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c372dafd1ef44c229e477a89d06db22a 2023-03-31 18:56:05,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,451 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,451 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:05,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:05,452 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:05,452 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:05,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c372dafd1ef44c229e477a89d06db22a, entries=200, sequenceid=496, filesize=14.3 K 2023-03-31 18:56:05,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/abae065691e348b6ad4bd1e5dc2e3850 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/abae065691e348b6ad4bd1e5dc2e3850 2023-03-31 18:56:05,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/abae065691e348b6ad4bd1e5dc2e3850, entries=150, sequenceid=496, filesize=11.9 K 2023-03-31 18:56:05,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/3d8fdecc991a4e13ab41914989f263c8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3d8fdecc991a4e13ab41914989f263c8 2023-03-31 18:56:05,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3d8fdecc991a4e13ab41914989f263c8, entries=150, sequenceid=496, filesize=11.9 K 2023-03-31 18:56:05,467 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 91eb2b257640c6018ad8eeccf0deb2cf in 463ms, sequenceid=496, compaction requested=true 2023-03-31 18:56:05,467 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:05,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:05,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:05,467 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:05,467 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:05,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:05,469 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54374 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:05,469 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:05,469 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:05,469 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:05,469 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,469 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,469 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/403fd6a8570848debc1d3997b4334b17, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5acb6d4c7a6c48d2a03e5e992919481d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/bea9dda17bd54f798dd7db20f84d24aa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c372dafd1ef44c229e477a89d06db22a] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=53.1 K 2023-03-31 18:56:05,469 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2705966f8f8e4a2987a876e393eb64a5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/7fc3e3d7d8fe4445908ecefc8c2e856c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a59ff037f4444573a027b13a5c95f086, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/abae065691e348b6ad4bd1e5dc2e3850] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=48.3 K 2023-03-31 18:56:05,469 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 403fd6a8570848debc1d3997b4334b17, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680288963106 2023-03-31 18:56:05,470 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2705966f8f8e4a2987a876e393eb64a5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680288963106 2023-03-31 18:56:05,470 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 5acb6d4c7a6c48d2a03e5e992919481d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1680288963480 2023-03-31 18:56:05,470 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 7fc3e3d7d8fe4445908ecefc8c2e856c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1680288963480 2023-03-31 18:56:05,470 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting bea9dda17bd54f798dd7db20f84d24aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680288963557 2023-03-31 18:56:05,470 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a59ff037f4444573a027b13a5c95f086, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680288963557 2023-03-31 18:56:05,471 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c372dafd1ef44c229e477a89d06db22a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680288964648 2023-03-31 18:56:05,471 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting abae065691e348b6ad4bd1e5dc2e3850, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680288964672 2023-03-31 18:56:05,481 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#453 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:05,484 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#454 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:05,501 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/ffe9a6537c8e455dbd8aec6672c96b13 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ffe9a6537c8e455dbd8aec6672c96b13 2023-03-31 18:56:05,506 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into ffe9a6537c8e455dbd8aec6672c96b13(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:05,506 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:05,506 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=12, startTime=1680288965467; duration=0sec 2023-03-31 18:56:05,506 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:05,506 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:05,509 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:05,509 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:05,509 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,509 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/636a672236a3438abf015cea5d6380ce, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/ecc0d9fefffd4a24b0a5f4de3b12bb32, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/15a1772f87dd4c19a5b616cd7df9b453, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3d8fdecc991a4e13ab41914989f263c8] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=48.3 K 2023-03-31 18:56:05,511 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 636a672236a3438abf015cea5d6380ce, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680288963106 2023-03-31 18:56:05,511 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ecc0d9fefffd4a24b0a5f4de3b12bb32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1680288963480 2023-03-31 18:56:05,511 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 15a1772f87dd4c19a5b616cd7df9b453, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680288963557 2023-03-31 18:56:05,512 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3d8fdecc991a4e13ab41914989f263c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680288964672 2023-03-31 18:56:05,526 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#455 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:05,542 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/5db43d7b83c24c2b8f06b3b478de635f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5db43d7b83c24c2b8f06b3b478de635f 2023-03-31 18:56:05,545 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into 5db43d7b83c24c2b8f06b3b478de635f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:05,545 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:05,546 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=12, startTime=1680288965467; duration=0sec 2023-03-31 18:56:05,546 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:05,552 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:05,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:05,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:05,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:05,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:05,555 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:05,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:05,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:05,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,556 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:05,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:05,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:05,558 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-31 18:56:05,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:05,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:05,559 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:05,559 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:05,559 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:05,559 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:05,579 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/3c7174f1baac4c8a890d8a4a8828c26c 2023-03-31 18:56:05,603 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/bf1b44b092114f5ea721f47875a1da0e 2023-03-31 18:56:05,622 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/8fa9c204fc324974aba480c62a03bc97 2023-03-31 18:56:05,626 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/3c7174f1baac4c8a890d8a4a8828c26c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/3c7174f1baac4c8a890d8a4a8828c26c 2023-03-31 18:56:05,632 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/3c7174f1baac4c8a890d8a4a8828c26c, entries=150, sequenceid=512, filesize=11.9 K 2023-03-31 18:56:05,633 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/bf1b44b092114f5ea721f47875a1da0e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/bf1b44b092114f5ea721f47875a1da0e 2023-03-31 18:56:05,636 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/bf1b44b092114f5ea721f47875a1da0e, entries=150, sequenceid=512, filesize=11.9 K 2023-03-31 18:56:05,637 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/8fa9c204fc324974aba480c62a03bc97 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8fa9c204fc324974aba480c62a03bc97 2023-03-31 18:56:05,641 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8fa9c204fc324974aba480c62a03bc97, entries=150, sequenceid=512, filesize=11.9 K 2023-03-31 18:56:05,641 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 91eb2b257640c6018ad8eeccf0deb2cf in 83ms, sequenceid=512, compaction requested=false 2023-03-31 18:56:05,641 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:05,641 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:05,643 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,644 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,644 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:05,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:05,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301be585[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:05,647 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:05,647 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,651 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:05,651 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:05,651 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:05,651 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:05,652 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:05,652 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,652 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:05,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:05,657 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:05,657 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:05,657 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:05,657 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:05,657 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:05,658 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,658 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,659 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,661 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,662 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,664 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,665 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:05,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:05,665 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:05,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,665 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:05,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:05,716 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:56:05,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:05,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:05,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:05,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:05,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:05,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:05,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680288965746, max timestamp=1680288965762Num uniques 859; 2023-03-31 18:56:05,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680288965746, max timestamp=1680288965762Num uniques 859; 2023-03-31 18:56:05,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:56:05,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680288965746, max timestamp=1680288965762Num uniques 859; 2023-03-31 18:56:05,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:56:05,765 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:05,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:56:05,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:05,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:05,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:05,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:05,768 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:05,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:05,768 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:05,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,775 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:05,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:05,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:05,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:05,779 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,779 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:05,780 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:05,780 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,780 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:05,780 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:05,780 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:05,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:05,782 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ded5bb6[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:05,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:05,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,786 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:05,786 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,786 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:05,787 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:05,788 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,788 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:05,787 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:05,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:05,790 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:05,790 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:05,790 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:05,791 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:05,791 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:05,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,795 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,797 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,799 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,799 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,799 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:05,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:05,799 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,800 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:05,800 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:05,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,800 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289025811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289025812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289025816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289025816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,900 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:05,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:05,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:05,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:05,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:05,906 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:05,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:05,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:05,906 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/ec20f74c0fdc47769b1310fad8c5d601 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ec20f74c0fdc47769b1310fad8c5d601 2023-03-31 18:56:05,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,908 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:05,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:05,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:05,910 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,910 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:05,910 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:05,910 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:05,911 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:05,911 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,911 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:05,911 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:05,911 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into ec20f74c0fdc47769b1310fad8c5d601(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:05,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,913 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,913 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:05,913 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:05,913 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=12, startTime=1680288965467; duration=0sec 2023-03-31 18:56:05,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,913 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:05,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:05,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2410cb4e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:05,916 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:05,916 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,918 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:05,918 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,919 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:05,919 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,919 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,920 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,920 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:05,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:05,925 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:05,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:05,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:05,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:05,926 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:05,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289025922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:05,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:05,929 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,930 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:05,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289025926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289025926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:05,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289025927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,933 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:05,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:05,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:05,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:05,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:05,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:05,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:05,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:05,934 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:05,934 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:05,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:05,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:05,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:05,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:06,034 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:06,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:06,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:06,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:06,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:06,038 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:06,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:06,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:06,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,041 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:06,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:06,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:06,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:06,042 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:06,042 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,042 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:06,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:06,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:06,044 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,044 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:06,044 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:06,044 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:06,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:06,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:06,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:06,047 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:06,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c80b78c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:06,049 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:06,049 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:06,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:06,050 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:06,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:06,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,054 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:06,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:06,055 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:06,055 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:06,055 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:06,055 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:06,055 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:06,055 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:06,055 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:06,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:06,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,057 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,061 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:06,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:06,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:06,061 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:06,061 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:06,062 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,062 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,062 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,062 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,062 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,062 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,062 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,063 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,063 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:06,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:06,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:06,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289026130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289026133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/0f7bf7d6aeb04cad8152ccbdbd5e8745 2023-03-31 18:56:06,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289026134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289026135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,162 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:06,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:06,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:06,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:06,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:06,165 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:06,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:06,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:06,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,165 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,167 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,167 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,167 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,167 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:06,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:06,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:06,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:06,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:06,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:06,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:06,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:06,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:06,168 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,168 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:06,168 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:06,168 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:06,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:06,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:06,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:06,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,171 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:06,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:06,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63585c54[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:06,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:06,173 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,174 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:06,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:06,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:06,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,175 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:06,176 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:06,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:06,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:06,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:06,180 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:06,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:06,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:06,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:06,180 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:06,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:06,182 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,183 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:06,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:06,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,186 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:06,186 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:06,186 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:06,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:06,286 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:06,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:06,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:06,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:06,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:06,290 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:06,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:06,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:06,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,292 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,293 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,293 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,293 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:06,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:06,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:06,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:06,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:06,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:06,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:06,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:06,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,295 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:06,295 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:06,295 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:06,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:06,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:06,295 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:06,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,296 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:06,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@646f8b62[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:06,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:06,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,299 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:06,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:06,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:06,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:06,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:06,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:06,302 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:06,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:06,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:06,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:06,302 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:06,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,304 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,304 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,306 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:06,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:06,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:06,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,309 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,309 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,308 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:06,309 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:06,309 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:06,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:06,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:06,409 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:06,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:06,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:06,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:06,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:06,412 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:06,412 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,412 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:06,412 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:06,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,414 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,414 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,414 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,414 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:06,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:06,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:06,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:06,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:06,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:06,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:06,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:06,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:06,416 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,416 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:06,416 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:06,416 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:06,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:06,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:06,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:06,420 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,420 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,421 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,421 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:06,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:06,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4607761a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:06,422 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:06,422 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,423 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,423 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:06,423 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:06,423 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:06,423 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:06,424 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:06,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:06,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,425 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:06,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:06,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:06,429 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:06,429 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:06,429 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:06,429 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:06,429 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:06,430 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,430 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,430 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,431 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:06,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:06,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:06,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:06,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:06,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:06,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,442 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,442 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:06,442 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:06,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289026438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289026441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289026442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289026443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,542 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:06,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:06,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:06,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:06,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:06,545 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:06,546 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,546 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:06,546 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:06,546 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,546 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,547 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/163b742d11c4485c93dfc0b0fefe8ac2 2023-03-31 18:56:06,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:06,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:06,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:06,549 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:06,549 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,549 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:06,552 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,552 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:06,552 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:06,552 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:06,554 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,554 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,555 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,555 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fb29581[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:06,556 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:06,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,559 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:06,559 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:06,559 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:06,559 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:06,562 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:06,562 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:06,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,562 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:06,564 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:06,564 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:06,564 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:06,564 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:06,564 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:06,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:06,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,570 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:06,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,573 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,581 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,581 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:06,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/64a229e92d414f24811a8ca6ffc92467 2023-03-31 18:56:06,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:06,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:06,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:06,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:06,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:06,583 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:06,583 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:06,587 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/0f7bf7d6aeb04cad8152ccbdbd5e8745 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0f7bf7d6aeb04cad8152ccbdbd5e8745 2023-03-31 18:56:06,593 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0f7bf7d6aeb04cad8152ccbdbd5e8745, entries=150, sequenceid=522, filesize=11.9 K 2023-03-31 18:56:06,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/163b742d11c4485c93dfc0b0fefe8ac2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/163b742d11c4485c93dfc0b0fefe8ac2 2023-03-31 18:56:06,598 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/163b742d11c4485c93dfc0b0fefe8ac2, entries=150, sequenceid=522, filesize=11.9 K 2023-03-31 18:56:06,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/64a229e92d414f24811a8ca6ffc92467 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/64a229e92d414f24811a8ca6ffc92467 2023-03-31 18:56:06,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/64a229e92d414f24811a8ca6ffc92467, entries=150, sequenceid=522, filesize=11.9 K 2023-03-31 18:56:06,606 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 91eb2b257640c6018ad8eeccf0deb2cf in 890ms, sequenceid=522, compaction requested=true 2023-03-31 18:56:06,606 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:06,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:06,606 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:06,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:06,606 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:06,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:06,608 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:06,608 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:06,608 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,609 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ffe9a6537c8e455dbd8aec6672c96b13, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/bf1b44b092114f5ea721f47875a1da0e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/163b742d11c4485c93dfc0b0fefe8ac2] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.6 K 2023-03-31 18:56:06,609 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:06,609 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:06,609 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,609 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ec20f74c0fdc47769b1310fad8c5d601, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/3c7174f1baac4c8a890d8a4a8828c26c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0f7bf7d6aeb04cad8152ccbdbd5e8745] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.6 K 2023-03-31 18:56:06,609 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ffe9a6537c8e455dbd8aec6672c96b13, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680288964672 2023-03-31 18:56:06,613 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting ec20f74c0fdc47769b1310fad8c5d601, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680288964672 2023-03-31 18:56:06,613 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting bf1b44b092114f5ea721f47875a1da0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680288965063 2023-03-31 18:56:06,614 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 3c7174f1baac4c8a890d8a4a8828c26c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680288965063 2023-03-31 18:56:06,615 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 163b742d11c4485c93dfc0b0fefe8ac2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1680288965710 2023-03-31 18:56:06,616 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 0f7bf7d6aeb04cad8152ccbdbd5e8745, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1680288965710 2023-03-31 18:56:06,622 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#462 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:06,625 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#463 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:06,638 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/c783ced293654173b020cccf6050c782 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c783ced293654173b020cccf6050c782 2023-03-31 18:56:06,643 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into c783ced293654173b020cccf6050c782(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:06,644 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:06,644 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=13, startTime=1680288966606; duration=0sec 2023-03-31 18:56:06,645 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:06,645 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:06,645 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:06,645 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:06,646 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,646 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5db43d7b83c24c2b8f06b3b478de635f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8fa9c204fc324974aba480c62a03bc97, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/64a229e92d414f24811a8ca6ffc92467] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.6 K 2023-03-31 18:56:06,646 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 5db43d7b83c24c2b8f06b3b478de635f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680288964672 2023-03-31 18:56:06,646 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8fa9c204fc324974aba480c62a03bc97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680288965063 2023-03-31 18:56:06,646 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 64a229e92d414f24811a8ca6ffc92467, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1680288965710 2023-03-31 18:56:06,652 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#464 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:06,665 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/0084edf3cdcb4eeab9d3559c1c07535f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/0084edf3cdcb4eeab9d3559c1c07535f 2023-03-31 18:56:06,668 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into 0084edf3cdcb4eeab9d3559c1c07535f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:06,668 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:06,668 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=13, startTime=1680288966606; duration=0sec 2023-03-31 18:56:06,668 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:06,683 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:06,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:06,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:06,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:06,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:06,687 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:06,687 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,687 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:06,687 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:06,687 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,688 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,689 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:06,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:06,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:06,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:06,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:06,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:06,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:06,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:06,690 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:06,690 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:06,690 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:06,692 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-31 18:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:06,693 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:06,713 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/6db2f1caeb4c402f8831732fede1ed23 2023-03-31 18:56:06,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:06,950 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:06,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289026957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289026958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289026962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:06,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:06,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289026963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,038 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/0fb3671dede0450ca10b80a85911e313 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/0fb3671dede0450ca10b80a85911e313 2023-03-31 18:56:07,042 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into 0fb3671dede0450ca10b80a85911e313(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:07,042 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:07,042 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=13, startTime=1680288966606; duration=0sec 2023-03-31 18:56:07,042 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:07,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289027066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289027066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289027067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289027070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,141 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/ed0ed0a23de84ccf9cfee1a928bfbf11 2023-03-31 18:56:07,152 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/4e6f8141edad46e8903a9a5ad11ce77f 2023-03-31 18:56:07,159 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/6db2f1caeb4c402f8831732fede1ed23 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/6db2f1caeb4c402f8831732fede1ed23 2023-03-31 18:56:07,163 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/6db2f1caeb4c402f8831732fede1ed23, entries=150, sequenceid=560, filesize=11.9 K 2023-03-31 18:56:07,167 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/ed0ed0a23de84ccf9cfee1a928bfbf11 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed0ed0a23de84ccf9cfee1a928bfbf11 2023-03-31 18:56:07,172 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed0ed0a23de84ccf9cfee1a928bfbf11, entries=150, sequenceid=560, filesize=11.9 K 2023-03-31 18:56:07,172 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/4e6f8141edad46e8903a9a5ad11ce77f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4e6f8141edad46e8903a9a5ad11ce77f 2023-03-31 18:56:07,175 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4e6f8141edad46e8903a9a5ad11ce77f, entries=150, sequenceid=560, filesize=11.9 K 2023-03-31 18:56:07,176 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 91eb2b257640c6018ad8eeccf0deb2cf in 484ms, sequenceid=560, compaction requested=false 2023-03-31 18:56:07,176 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:07,176 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:07,176 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:07,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:07,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:07,177 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:07,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:07,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:07,180 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:07,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:07,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea9cadf[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:07,182 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:07,182 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,184 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,185 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:07,185 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:07,185 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:07,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:07,186 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,186 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:07,186 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:07,186 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:07,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:07,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:07,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:07,190 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:07,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:07,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:07,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:07,190 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:07,191 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,191 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:07,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,192 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:07,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:07,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:07,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:07,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:07,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:07,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:07,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:07,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:07,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,200 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:07,200 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:07,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:07,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:07,276 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:56:07,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:07,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:07,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:07,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:07,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:07,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:07,285 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/1d10325cc3c04276a9b585af97fe324b 2023-03-31 18:56:07,295 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/95fdc54a0baa4e6b852da08831e5c5b4 2023-03-31 18:56:07,301 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:07,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/7f7a251540e14da8ab9ff81f242aead5 2023-03-31 18:56:07,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:07,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289027312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289027313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288967300, max timestamp=1680288967315Num uniques 859; 2023-03-31 18:56:07,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:07,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288967300, max timestamp=1680288967315Num uniques 859; 2023-03-31 18:56:07,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:07,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288967300, max timestamp=1680288967315Num uniques 859; 2023-03-31 18:56:07,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:07,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:07,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:07,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:07,324 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:07,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:07,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:07,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:07,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/1d10325cc3c04276a9b585af97fe324b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1d10325cc3c04276a9b585af97fe324b 2023-03-31 18:56:07,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:07,327 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:07,327 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,327 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:07,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:07,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:07,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:07,332 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:07,332 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:07,332 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:07,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1d10325cc3c04276a9b585af97fe324b, entries=150, sequenceid=572, filesize=11.9 K 2023-03-31 18:56:07,333 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:07,333 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:07,333 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:07,333 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:07,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/95fdc54a0baa4e6b852da08831e5c5b4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/95fdc54a0baa4e6b852da08831e5c5b4 2023-03-31 18:56:07,335 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,335 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:07,336 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,336 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:07,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:07,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768568f3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:07,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:07,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/95fdc54a0baa4e6b852da08831e5c5b4, entries=150, sequenceid=572, filesize=11.9 K 2023-03-31 18:56:07,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:07,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/7f7a251540e14da8ab9ff81f242aead5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/7f7a251540e14da8ab9ff81f242aead5 2023-03-31 18:56:07,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:07,342 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:07,343 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:07,343 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:07,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:07,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/7f7a251540e14da8ab9ff81f242aead5, entries=150, sequenceid=572, filesize=11.9 K 2023-03-31 18:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,346 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=80.51 KB/82440 for 91eb2b257640c6018ad8eeccf0deb2cf in 70ms, sequenceid=572, compaction requested=true 2023-03-31 18:56:07,346 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:07,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:07,346 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:07,346 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:07,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:07,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:07,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:07,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:07,346 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:07,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:07,346 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:07,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:07,347 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:07,347 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:07,347 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:07,347 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c783ced293654173b020cccf6050c782, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/6db2f1caeb4c402f8831732fede1ed23, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1d10325cc3c04276a9b585af97fe324b] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.7 K 2023-03-31 18:56:07,348 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c783ced293654173b020cccf6050c782, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1680288965710 2023-03-31 18:56:07,348 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:07,348 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:07,348 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:07,348 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 6db2f1caeb4c402f8831732fede1ed23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680288965810 2023-03-31 18:56:07,348 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/0fb3671dede0450ca10b80a85911e313, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed0ed0a23de84ccf9cfee1a928bfbf11, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/95fdc54a0baa4e6b852da08831e5c5b4] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.7 K 2023-03-31 18:56:07,348 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 1d10325cc3c04276a9b585af97fe324b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680288966957 2023-03-31 18:56:07,348 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,348 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,349 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 0fb3671dede0450ca10b80a85911e313, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1680288965710 2023-03-31 18:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:07,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:07,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:07,349 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,349 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ed0ed0a23de84ccf9cfee1a928bfbf11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680288965810 2023-03-31 18:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:07,350 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 95fdc54a0baa4e6b852da08831e5c5b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680288966957 2023-03-31 18:56:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:07,351 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:07,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=87.22 KB heapSize=240.84 KB 2023-03-31 18:56:07,357 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:07,357 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:07,357 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:07,357 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:07,357 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:07,357 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,357 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:07,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:07,358 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#471 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:07,358 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:07,358 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:07,357 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:07,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:07,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:07,358 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#472 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:07,358 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,358 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,358 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,358 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,358 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,358 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:07,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:07,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:07,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:07,391 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/56b34c314eca49cda2c1ed865329598c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/56b34c314eca49cda2c1ed865329598c 2023-03-31 18:56:07,395 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into 56b34c314eca49cda2c1ed865329598c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:07,395 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:07,395 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=13, startTime=1680288967346; duration=0sec 2023-03-31 18:56:07,395 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:07,395 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:07,397 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:07,397 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:07,397 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:07,397 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/0084edf3cdcb4eeab9d3559c1c07535f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4e6f8141edad46e8903a9a5ad11ce77f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/7f7a251540e14da8ab9ff81f242aead5] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.7 K 2023-03-31 18:56:07,397 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 0084edf3cdcb4eeab9d3559c1c07535f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1680288965710 2023-03-31 18:56:07,397 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4e6f8141edad46e8903a9a5ad11ce77f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680288965810 2023-03-31 18:56:07,397 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 7f7a251540e14da8ab9ff81f242aead5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680288966957 2023-03-31 18:56:07,408 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/74fd72c1edc1409db372e1452947de72 2023-03-31 18:56:07,409 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:07,416 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/e3209507cbd44250a3caa4a565be8b9d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e3209507cbd44250a3caa4a565be8b9d 2023-03-31 18:56:07,422 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into e3209507cbd44250a3caa4a565be8b9d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:07,422 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:07,422 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=13, startTime=1680288967347; duration=0sec 2023-03-31 18:56:07,422 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:07,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289027418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289027419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/63d7d14743854db9aaae8dbfb76ec10e 2023-03-31 18:56:07,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289027423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289027424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,437 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/78fb767f482b4659af486913b9dbd579 2023-03-31 18:56:07,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/74fd72c1edc1409db372e1452947de72 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/74fd72c1edc1409db372e1452947de72 2023-03-31 18:56:07,444 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/74fd72c1edc1409db372e1452947de72, entries=200, sequenceid=602, filesize=14.3 K 2023-03-31 18:56:07,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/63d7d14743854db9aaae8dbfb76ec10e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/63d7d14743854db9aaae8dbfb76ec10e 2023-03-31 18:56:07,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/63d7d14743854db9aaae8dbfb76ec10e, entries=150, sequenceid=602, filesize=11.9 K 2023-03-31 18:56:07,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/78fb767f482b4659af486913b9dbd579 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/78fb767f482b4659af486913b9dbd579 2023-03-31 18:56:07,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/78fb767f482b4659af486913b9dbd579, entries=150, sequenceid=602, filesize=11.9 K 2023-03-31 18:56:07,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~260.72 KB/266976, currentSize=87.22 KB/89310 for 91eb2b257640c6018ad8eeccf0deb2cf in 99ms, sequenceid=602, compaction requested=false 2023-03-31 18:56:07,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:07,458 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:07,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:07,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:07,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:07,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:07,462 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:07,462 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:07,462 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:07,462 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:07,462 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,462 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,465 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:07,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:07,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:07,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:07,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:07,466 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:07,466 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:07,466 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:07,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:07,468 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:07,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:07,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:07,468 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-31 18:56:07,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:07,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:07,468 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:07,469 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:07,469 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:07,469 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:07,483 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/d1d2dc836d794ce7bb08305cad6839f8 2023-03-31 18:56:07,502 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/4e8b49f823004086a8e1020b1581086f 2023-03-31 18:56:07,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:07,535 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:07,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289027568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289027569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289027625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289027626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289027675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289027677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,782 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/0070f1d2e30b48a4bcb9427e2e8e7f3f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0070f1d2e30b48a4bcb9427e2e8e7f3f 2023-03-31 18:56:07,786 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into 0070f1d2e30b48a4bcb9427e2e8e7f3f(size=13.0 K), total size for store is 27.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:07,787 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:07,787 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=13, startTime=1680288967346; duration=0sec 2023-03-31 18:56:07,787 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:07,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289027881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289027882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,915 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/5244e0fba4b94047827db8e8feb6b2e5 2023-03-31 18:56:07,919 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/d1d2dc836d794ce7bb08305cad6839f8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d1d2dc836d794ce7bb08305cad6839f8 2023-03-31 18:56:07,925 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d1d2dc836d794ce7bb08305cad6839f8, entries=150, sequenceid=620, filesize=11.9 K 2023-03-31 18:56:07,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/4e8b49f823004086a8e1020b1581086f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4e8b49f823004086a8e1020b1581086f 2023-03-31 18:56:07,930 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4e8b49f823004086a8e1020b1581086f, entries=150, sequenceid=620, filesize=11.9 K 2023-03-31 18:56:07,930 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/5244e0fba4b94047827db8e8feb6b2e5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5244e0fba4b94047827db8e8feb6b2e5 2023-03-31 18:56:07,934 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5244e0fba4b94047827db8e8feb6b2e5, entries=150, sequenceid=620, filesize=11.9 K 2023-03-31 18:56:07,934 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 91eb2b257640c6018ad8eeccf0deb2cf in 466ms, sequenceid=620, compaction requested=true 2023-03-31 18:56:07,934 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:07,934 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:07,934 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:07,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:07,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:07,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:07,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:07,936 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-31 18:56:07,936 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,937 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:07,937 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,937 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:07,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:07,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:07,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:07,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:07,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:07,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:07,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:07,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:07,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:07,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b4f779f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:07,939 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:07,939 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,940 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:07,941 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,941 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:07,942 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:07,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:07,945 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:07,945 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:07,945 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:07,945 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:07,945 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:07,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:07,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:07,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:07,950 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/78f36b676735420e904bc0e589901276 2023-03-31 18:56:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:07,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,954 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,954 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:07,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:07,954 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:07,955 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:07,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:07,954 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:07,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:07,954 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:07,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:07,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:07,955 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:07,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:07,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289027987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:07,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:07,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289027987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,055 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:08,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:08,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:08,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:08,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:08,058 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:08,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:08,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:08,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,059 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,060 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,060 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:08,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:08,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:08,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:08,061 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,061 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:08,061 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:08,062 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,062 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:08,062 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:08,062 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,062 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:08,062 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:08,063 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@500af5ea[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:08,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:08,065 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,068 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,068 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,068 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:08,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:08,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:08,068 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:08,069 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,069 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:08,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:08,072 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:08,072 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:08,072 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:08,072 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:08,072 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:08,073 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,073 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,073 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,075 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,078 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,078 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,078 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:08,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:08,078 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,078 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,079 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,079 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:08,079 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,079 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,079 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,079 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,079 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:08,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:08,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289028099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:08,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289028099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,185 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:08,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:08,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:08,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:08,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:08,189 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:08,189 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,189 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:08,189 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:08,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:08,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289028186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,189 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,191 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,191 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:08,193 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,193 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:08,193 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:08,193 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:08,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:08,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,195 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:08,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289028189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d53206[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:08,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,198 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,198 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:08,198 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:08,198 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:08,198 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,198 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:08,199 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,199 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,199 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:08,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:08,202 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:08,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:08,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:08,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:08,202 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:08,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,204 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,208 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,208 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,208 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:08,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:08,216 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:08,216 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:08,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:08,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:08,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289028310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:08,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289028311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,317 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:08,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:08,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:08,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:08,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:08,320 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:08,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:08,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:08,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,324 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:08,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:08,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:08,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:08,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:08,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:08,327 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,327 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:08,327 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:08,327 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,330 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4a0e1f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:08,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:08,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,333 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:08,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:08,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:08,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,334 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,334 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:08,337 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:08,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:08,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:08,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:08,337 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:08,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,339 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,347 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,347 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,347 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,347 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:08,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:08,347 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,347 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,348 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:08,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,348 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,348 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:08,348 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,348 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:08,348 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/03accf55ca7e452dae5f2894b8c85ffd 2023-03-31 18:56:08,448 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:08,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:08,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:08,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:08,452 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:08,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:08,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:08,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,453 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,454 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,454 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:08,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:08,455 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,455 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:08,455 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:08,455 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:08,456 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,456 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:08,456 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:08,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,458 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,458 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,458 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dc977a6[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:08,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:08,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,460 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:08,460 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,461 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:08,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,463 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,463 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,463 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:08,465 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:08,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:08,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:08,465 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:08,465 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:08,467 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,467 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,467 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,467 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,468 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,471 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,471 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,471 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,471 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,471 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,471 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,471 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,471 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,471 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:08,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:08,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:08,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,471 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:08,471 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:08,571 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:08,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:08,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:08,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:08,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:08,576 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:08,576 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,576 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:08,576 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:08,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,578 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:08,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:08,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:08,583 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a6b7c50[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:08,584 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:08,585 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:08,586 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,586 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:08,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:08,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:08,590 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:08,591 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:08,591 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:08,591 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:08,591 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:08,592 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,592 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,592 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,592 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,593 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,596 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:08,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:08,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,596 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:08,596 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:08,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:08,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:08,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289028617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:08,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289028617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,697 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:08,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:08,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:08,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289028694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:08,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:08,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:08,700 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:08,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:08,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:08,701 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,701 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,702 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,702 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,702 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,702 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:08,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289028699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:08,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:08,705 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,705 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:08,705 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:08,705 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:08,707 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,708 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,708 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,708 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f73019[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:08,709 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:08,709 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,714 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:08,715 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:08,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,715 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:08,716 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,716 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,716 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,716 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:08,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:08,718 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:08,719 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:08,719 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:08,719 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:08,719 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:08,720 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,720 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,720 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:08,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:08,721 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,722 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:08,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:08,724 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:08,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:08,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,725 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:08,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,725 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:08,725 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:08,725 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,725 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:08,725 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:08,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/65f52070ab26437cabe83ad111401412 2023-03-31 18:56:08,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/78f36b676735420e904bc0e589901276 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78f36b676735420e904bc0e589901276 2023-03-31 18:56:08,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78f36b676735420e904bc0e589901276, entries=200, sequenceid=639, filesize=14.3 K 2023-03-31 18:56:08,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/03accf55ca7e452dae5f2894b8c85ffd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03accf55ca7e452dae5f2894b8c85ffd 2023-03-31 18:56:08,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03accf55ca7e452dae5f2894b8c85ffd, entries=150, sequenceid=639, filesize=11.9 K 2023-03-31 18:56:08,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/65f52070ab26437cabe83ad111401412 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/65f52070ab26437cabe83ad111401412 2023-03-31 18:56:08,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/65f52070ab26437cabe83ad111401412, entries=150, sequenceid=639, filesize=11.9 K 2023-03-31 18:56:08,786 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 91eb2b257640c6018ad8eeccf0deb2cf in 850ms, sequenceid=639, compaction requested=true 2023-03-31 18:56:08,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:08,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:08,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 18:56:08,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-31 18:56:08,787 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:08,787 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:08,788 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54714 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:08,788 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:08,788 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,788 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0070f1d2e30b48a4bcb9427e2e8e7f3f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/74fd72c1edc1409db372e1452947de72, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d1d2dc836d794ce7bb08305cad6839f8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78f36b676735420e904bc0e589901276] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=53.4 K 2023-03-31 18:56:08,789 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:08,789 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:08,789 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,789 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e3209507cbd44250a3caa4a565be8b9d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/78fb767f482b4659af486913b9dbd579, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5244e0fba4b94047827db8e8feb6b2e5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/65f52070ab26437cabe83ad111401412] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=48.7 K 2023-03-31 18:56:08,789 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 0070f1d2e30b48a4bcb9427e2e8e7f3f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680288966957 2023-03-31 18:56:08,789 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e3209507cbd44250a3caa4a565be8b9d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680288966957 2023-03-31 18:56:08,790 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 74fd72c1edc1409db372e1452947de72, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1680288967330 2023-03-31 18:56:08,790 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 78fb767f482b4659af486913b9dbd579, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1680288967343 2023-03-31 18:56:08,790 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d1d2dc836d794ce7bb08305cad6839f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680288967382 2023-03-31 18:56:08,790 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 5244e0fba4b94047827db8e8feb6b2e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680288967382 2023-03-31 18:56:08,791 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 78f36b676735420e904bc0e589901276, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1680288967561 2023-03-31 18:56:08,791 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 65f52070ab26437cabe83ad111401412, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1680288967567 2023-03-31 18:56:08,798 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#483 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:08,804 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#484 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:08,825 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:08,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:08,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:08,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:08,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:08,828 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:08,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:08,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:08,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,832 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:08,832 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:08,833 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:08,833 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:08,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:08,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:08,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:08,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:08,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:08,835 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:08,835 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:08,835 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:08,835 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:08,835 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-31 18:56:08,836 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:08,836 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:08,836 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:08,836 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:08,836 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:08,836 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:09,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:09,129 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:09,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289029170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289029174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,207 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/5afa97edeaaf4e93b052b9cccfc1692b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5afa97edeaaf4e93b052b9cccfc1692b 2023-03-31 18:56:09,212 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into 5afa97edeaaf4e93b052b9cccfc1692b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:09,213 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:09,213 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=12, startTime=1680288968787; duration=0sec 2023-03-31 18:56:09,213 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/d139a2bf30354f8396c380aa05c58f27 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d139a2bf30354f8396c380aa05c58f27 2023-03-31 18:56:09,213 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:09,213 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:09,214 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:09,214 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:09,214 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:09,214 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/56b34c314eca49cda2c1ed865329598c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/63d7d14743854db9aaae8dbfb76ec10e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4e8b49f823004086a8e1020b1581086f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03accf55ca7e452dae5f2894b8c85ffd] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=48.7 K 2023-03-31 18:56:09,214 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 56b34c314eca49cda2c1ed865329598c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680288966957 2023-03-31 18:56:09,215 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 63d7d14743854db9aaae8dbfb76ec10e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1680288967343 2023-03-31 18:56:09,215 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 4e8b49f823004086a8e1020b1581086f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680288967382 2023-03-31 18:56:09,215 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 03accf55ca7e452dae5f2894b8c85ffd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1680288967567 2023-03-31 18:56:09,217 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into d139a2bf30354f8396c380aa05c58f27(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:09,217 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:09,217 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=12, startTime=1680288968787; duration=0sec 2023-03-31 18:56:09,217 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:09,223 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#486 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:09,235 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/92294d27f80246538923ff6bd43639c3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/92294d27f80246538923ff6bd43639c3 2023-03-31 18:56:09,238 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into 92294d27f80246538923ff6bd43639c3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:09,238 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:09,238 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=12, startTime=1680288968787; duration=0sec 2023-03-31 18:56:09,238 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:09,247 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/158a52b24c2946cc8f0808004ad4c7cf 2023-03-31 18:56:09,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289029276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289029279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289029482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289029484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,655 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/9f563e3ec6f74703938f632bbe62bce7 2023-03-31 18:56:09,664 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/720fe359cf974a5da7aaf67c65c63138 2023-03-31 18:56:09,667 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/158a52b24c2946cc8f0808004ad4c7cf as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/158a52b24c2946cc8f0808004ad4c7cf 2023-03-31 18:56:09,670 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/158a52b24c2946cc8f0808004ad4c7cf, entries=150, sequenceid=653, filesize=11.9 K 2023-03-31 18:56:09,671 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/9f563e3ec6f74703938f632bbe62bce7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f563e3ec6f74703938f632bbe62bce7 2023-03-31 18:56:09,674 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f563e3ec6f74703938f632bbe62bce7, entries=150, sequenceid=653, filesize=11.9 K 2023-03-31 18:56:09,675 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/720fe359cf974a5da7aaf67c65c63138 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/720fe359cf974a5da7aaf67c65c63138 2023-03-31 18:56:09,679 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/720fe359cf974a5da7aaf67c65c63138, entries=150, sequenceid=653, filesize=11.9 K 2023-03-31 18:56:09,679 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 91eb2b257640c6018ad8eeccf0deb2cf in 844ms, sequenceid=653, compaction requested=false 2023-03-31 18:56:09,679 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:09,679 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:09,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:09,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:09,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:09,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:09,681 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:09,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:09,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:09,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:09,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:09,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:09,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb0ec8e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:09,684 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:09,684 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,685 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,685 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:09,685 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:09,685 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:09,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:09,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:09,687 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:09,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:09,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:09,689 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:09,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:09,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:09,689 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:09,689 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:09,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:09,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:09,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:09,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:09,698 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:09,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:09,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:09,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,698 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:09,698 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:09,698 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:09,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:09,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:09,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:09,701 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-31 18:56:09,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:09,701 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:09,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:09,701 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:09,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:09,702 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:09,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/40a043b6c2644e4a9662a9abd37b5b23 2023-03-31 18:56:09,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289029738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289029743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289029790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289029792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,798 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:09,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:09,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:09,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:09,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:09,801 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:09,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:09,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:09,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:09,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,801 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,803 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,803 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:09,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:09,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:09,803 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,803 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:09,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:09,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:09,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:09,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:09,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:09,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:09,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:09,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:09,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:09,804 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:09,804 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:09,804 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:09,804 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:09,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:09,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:09,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:09,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:09,806 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:09,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:09,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:09,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:09,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:09,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:09,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68d2c0b9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:09,808 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:09,808 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,809 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,810 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:09,810 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:09,810 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:09,810 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,810 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:09,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,811 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:09,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:09,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:09,811 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:09,811 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:09,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:09,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:09,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:09,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:09,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:09,813 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:09,813 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:09,813 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:09,813 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:09,813 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:09,815 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,815 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:09,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,815 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:09,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,817 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:09,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:09,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:09,819 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:09,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:09,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:09,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:09,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,819 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:09,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,819 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:09,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:09,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289029845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:09,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289029845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,919 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:09,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:09,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:09,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:09,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:09,922 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:09,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:09,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:09,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:09,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:09,924 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:09,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:09,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:09,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:09,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:09,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:09,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:09,926 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:09,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,928 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:09,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:09,931 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,931 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:09,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:09,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@695aa80f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:09,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:09,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,934 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:09,934 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:09,934 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:09,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:09,939 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,939 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:09,939 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:09,939 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:09,942 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:09,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:09,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:09,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:09,942 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:09,944 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,944 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:09,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,944 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:09,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:09,944 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,945 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:09,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:09,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:09,948 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:09,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:09,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:09,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:09,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:09,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:09,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:09,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:09,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:09,949 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:09,949 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:10,049 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:10,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:10,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:10,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:10,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:10,052 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:10,052 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,052 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:10,052 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:10,052 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,052 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,055 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:10,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:10,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289030050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:10,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:10,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289030051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:10,056 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:10,057 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,057 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:10,057 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:10,057 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,057 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:10,057 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:10,057 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,057 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:10,057 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:10,060 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,060 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,060 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,060 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:10,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53173ac5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:10,062 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:10,062 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,063 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:10,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:10,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:10,063 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,063 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:10,065 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:10,065 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,065 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,065 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:10,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:10,067 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:10,067 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:10,067 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:10,067 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:10,067 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:10,070 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,070 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,070 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,070 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,071 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,074 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,074 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:10,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:10,075 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:10,075 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:10,075 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:10,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/a66d8161738c42909a4e9e28d5b629f2 2023-03-31 18:56:10,129 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/a460289a6f5a4028824d50a9acf09055 2023-03-31 18:56:10,134 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/40a043b6c2644e4a9662a9abd37b5b23 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40a043b6c2644e4a9662a9abd37b5b23 2023-03-31 18:56:10,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40a043b6c2644e4a9662a9abd37b5b23, entries=200, sequenceid=675, filesize=14.3 K 2023-03-31 18:56:10,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/a66d8161738c42909a4e9e28d5b629f2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a66d8161738c42909a4e9e28d5b629f2 2023-03-31 18:56:10,147 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a66d8161738c42909a4e9e28d5b629f2, entries=150, sequenceid=675, filesize=11.9 K 2023-03-31 18:56:10,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/a460289a6f5a4028824d50a9acf09055 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a460289a6f5a4028824d50a9acf09055 2023-03-31 18:56:10,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a460289a6f5a4028824d50a9acf09055, entries=150, sequenceid=675, filesize=11.9 K 2023-03-31 18:56:10,154 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 91eb2b257640c6018ad8eeccf0deb2cf in 453ms, sequenceid=675, compaction requested=true 2023-03-31 18:56:10,154 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:10,155 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:10,155 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:10,155 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:10,155 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:10,155 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:10,156 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40224 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:10,156 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:10,156 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:10,156 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,156 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:10,156 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5afa97edeaaf4e93b052b9cccfc1692b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/158a52b24c2946cc8f0808004ad4c7cf, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40a043b6c2644e4a9662a9abd37b5b23] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=39.3 K 2023-03-31 18:56:10,156 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,156 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/92294d27f80246538923ff6bd43639c3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f563e3ec6f74703938f632bbe62bce7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a66d8161738c42909a4e9e28d5b629f2] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=36.9 K 2023-03-31 18:56:10,156 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 5afa97edeaaf4e93b052b9cccfc1692b, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1680288967567 2023-03-31 18:56:10,156 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 92294d27f80246538923ff6bd43639c3, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1680288967567 2023-03-31 18:56:10,156 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 158a52b24c2946cc8f0808004ad4c7cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1680288967966 2023-03-31 18:56:10,157 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9f563e3ec6f74703938f632bbe62bce7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1680288967966 2023-03-31 18:56:10,157 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 40a043b6c2644e4a9662a9abd37b5b23, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1680288969162 2023-03-31 18:56:10,157 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a66d8161738c42909a4e9e28d5b629f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1680288969162 2023-03-31 18:56:10,167 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#492 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:10,168 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:10,175 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:10,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:10,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:10,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:10,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:10,180 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:10,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:10,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:10,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,182 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,183 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,183 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:10,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:10,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:10,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:10,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:10,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:10,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:10,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:10,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:10,195 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 18:56:10,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:10,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:10,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:10,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:10,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:10,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:10,218 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/e3a7831b49ae43bfadc852ad3cf00fd6 2023-03-31 18:56:10,233 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/15a52e6c304147929dad28578a82ad52 2023-03-31 18:56:10,253 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/f6201811b08a440fbfe413bbe1cb0ba3 2023-03-31 18:56:10,257 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/e3a7831b49ae43bfadc852ad3cf00fd6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/e3a7831b49ae43bfadc852ad3cf00fd6 2023-03-31 18:56:10,260 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/e3a7831b49ae43bfadc852ad3cf00fd6, entries=100, sequenceid=688, filesize=9.5 K 2023-03-31 18:56:10,261 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/15a52e6c304147929dad28578a82ad52 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/15a52e6c304147929dad28578a82ad52 2023-03-31 18:56:10,264 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/15a52e6c304147929dad28578a82ad52, entries=100, sequenceid=688, filesize=9.5 K 2023-03-31 18:56:10,266 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/f6201811b08a440fbfe413bbe1cb0ba3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/f6201811b08a440fbfe413bbe1cb0ba3 2023-03-31 18:56:10,269 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/f6201811b08a440fbfe413bbe1cb0ba3, entries=100, sequenceid=688, filesize=9.5 K 2023-03-31 18:56:10,270 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 91eb2b257640c6018ad8eeccf0deb2cf in 75ms, sequenceid=688, compaction requested=true 2023-03-31 18:56:10,270 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:10,270 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:10,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:10,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:10,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:10,272 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:10,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aba68b5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:10,274 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:10,274 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,275 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,275 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:10,275 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:10,275 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:10,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,276 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:10,279 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:10,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:10,281 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:10,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:10,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:10,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:10,282 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:10,283 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,283 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,283 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,285 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,289 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:10,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,290 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:10,290 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:10,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:10,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:56:10,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:10,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:10,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:10,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:10,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:10,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:10,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/934b15ed2c2444af802fb18c018abecd 2023-03-31 18:56:10,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680288970367, max timestamp=1680288970382Num uniques 859; 2023-03-31 18:56:10,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680288970367, max timestamp=1680288970382Num uniques 859; 2023-03-31 18:56:10,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 18:56:10,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680288970367, max timestamp=1680288970382Num uniques 859; 2023-03-31 18:56:10,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 18:56:10,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 18:56:10,390 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:10,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:10,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:10,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:10,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:10,395 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:10,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:10,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:10,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,395 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,396 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,396 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,397 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:10,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:10,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:10,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:10,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:10,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:10,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:10,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:10,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:10,398 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,398 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:10,398 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:10,398 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:10,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:10,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:10,398 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:10,401 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,401 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,402 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,402 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5413a253[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:10,403 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:10,403 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,404 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,405 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:10,405 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:10,405 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:10,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:10,406 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:10,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,406 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,406 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:10,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:10,409 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:10,409 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:10,409 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:10,409 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:10,409 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:10,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,411 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,419 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,419 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,419 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,419 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:10,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:10,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,419 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,420 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:10,420 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,420 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:10,420 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,420 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,420 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,420 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:10,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289030420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289030421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289030429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289030431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,520 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:10,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:10,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:10,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:10,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:10,523 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:10,523 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,523 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:10,523 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:10,524 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,524 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,525 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,525 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,526 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,526 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:10,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:10,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:10,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:10,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:10,527 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:10,527 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,527 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:10,527 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:10,527 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:10,527 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,527 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:10,528 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:10,528 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:10,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:10,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:10,528 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:10,530 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,530 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,531 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,531 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:10,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6835720e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:10,533 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:10,533 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,534 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:10,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:10,534 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,535 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:10,535 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:10,537 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,537 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:10,537 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,537 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289030532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289030532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289030537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:10,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:10,542 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:10,542 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:10,542 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:10,542 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:10,542 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:10,543 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,543 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,544 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,544 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289030537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,545 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:10,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:10,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:10,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,548 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:10,548 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:10,548 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,586 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/b33da7d897f74c409178c7a1c6e6d032 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/b33da7d897f74c409178c7a1c6e6d032 2023-03-31 18:56:10,587 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/43c9d0c41e714227a0f031f2aebb146d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43c9d0c41e714227a0f031f2aebb146d 2023-03-31 18:56:10,591 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into b33da7d897f74c409178c7a1c6e6d032(size=13.2 K), total size for store is 22.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:10,591 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:10,591 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=13, startTime=1680288970155; duration=0sec 2023-03-31 18:56:10,591 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:10,591 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:10,592 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into 43c9d0c41e714227a0f031f2aebb146d(size=13.2 K), total size for store is 22.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:10,592 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:10,593 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=13, startTime=1680288970155; duration=0sec 2023-03-31 18:56:10,593 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:10,593 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47526 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:10,593 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:10,593 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,593 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d139a2bf30354f8396c380aa05c58f27, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/720fe359cf974a5da7aaf67c65c63138, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a460289a6f5a4028824d50a9acf09055, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/f6201811b08a440fbfe413bbe1cb0ba3] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=46.4 K 2023-03-31 18:56:10,593 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d139a2bf30354f8396c380aa05c58f27, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1680288967567 2023-03-31 18:56:10,594 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 720fe359cf974a5da7aaf67c65c63138, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1680288967966 2023-03-31 18:56:10,594 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a460289a6f5a4028824d50a9acf09055, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1680288969162 2023-03-31 18:56:10,594 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f6201811b08a440fbfe413bbe1cb0ba3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1680288969733 2023-03-31 18:56:10,601 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#499 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:10,609 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/2c0f06e25eaa4c9d98bc649421ad8c77 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/2c0f06e25eaa4c9d98bc649421ad8c77 2023-03-31 18:56:10,613 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into 2c0f06e25eaa4c9d98bc649421ad8c77(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:10,613 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:10,613 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=12, startTime=1680288970155; duration=0sec 2023-03-31 18:56:10,613 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:10,648 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:10,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:10,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:10,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:10,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:10,652 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:10,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:10,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:10,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,653 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,653 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,654 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,654 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:10,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:10,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:10,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:10,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:10,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:10,655 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:10,656 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,656 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:10,656 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:10,656 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:10,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:10,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:10,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:10,659 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,659 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,659 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,659 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:10,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:10,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ebcdf[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:10,661 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:10,661 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,662 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:10,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:10,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:10,663 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,663 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:10,664 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,664 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:10,664 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,664 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:10,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:10,667 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:10,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:10,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:10,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:10,667 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:10,668 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,668 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,669 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,669 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,670 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,674 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,674 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:10,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,675 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,675 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:10,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:10,676 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:10,676 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:10,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289030739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289030740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289030742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/4c9b748f11d541cd806a5a0cd68b0003 2023-03-31 18:56:10,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:10,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289030747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,776 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:10,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:10,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:10,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:10,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:10,779 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:10,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:10,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:10,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,781 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,781 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,781 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:10,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:10,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:10,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:10,784 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,784 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:10,784 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:10,784 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:10,786 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:10,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b1893f9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:10,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:10,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,791 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:10,791 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:10,791 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,791 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:10,791 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:10,797 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,797 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:10,797 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,797 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:10,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:10,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:10,800 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:10,800 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:10,800 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:10,800 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:10,800 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:10,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,803 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,804 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,809 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,809 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,809 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,809 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:10,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:10,809 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,809 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,809 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,809 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,810 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:10,810 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:10,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,810 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,910 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:10,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:10,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:10,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:10,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:10,913 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:10,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:10,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:10,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,914 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,916 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:10,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:10,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:10,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:10,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:10,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:10,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:10,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:10,918 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,918 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:10,918 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:10,918 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:10,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:10,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:10,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:10,919 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:10,920 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,920 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,920 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,921 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:10,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:10,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3250b7f3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:10,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:10,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,923 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,923 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:10,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:10,924 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:10,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:10,926 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:10,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:10,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:10,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:10,930 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:10,930 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:10,930 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:10,930 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:10,930 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:10,933 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,933 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,933 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:10,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:10,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,935 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,937 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:10,937 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:10,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:10,937 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:10,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:10,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:10,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:10,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:10,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,938 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:10,938 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:10,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:10,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:10,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:10,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:11,038 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:11,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:11,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:11,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:11,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:11,041 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:11,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:11,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:11,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:11,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,043 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:11,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:11,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:11,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:11,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:11,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:11,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:11,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:11,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:11,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:11,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:11,046 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:11,046 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:11,046 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:11,046 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:11,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,046 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:11,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289031046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,047 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:11,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:11,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:11,047 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,048 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:11,048 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:11,049 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,049 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:11,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:11,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71b606b5[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:11,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:11,051 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289031046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289031046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,053 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,053 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:11,053 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:11,053 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:11,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:11,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:11,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:11,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,055 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:11,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:11,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:11,058 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:11,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:11,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:11,058 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:11,058 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:11,059 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,059 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:11,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,059 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:11,060 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289031053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,061 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:11,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:11,064 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:11,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:11,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:11,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:11,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:11,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,064 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:11,064 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:11,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:11,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,064 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:11,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/da93c37f9eb24dd3a0b794c4fb3b28a5 2023-03-31 18:56:11,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/934b15ed2c2444af802fb18c018abecd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/934b15ed2c2444af802fb18c018abecd 2023-03-31 18:56:11,164 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:11,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:11,165 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/934b15ed2c2444af802fb18c018abecd, entries=150, sequenceid=698, filesize=11.9 K 2023-03-31 18:56:11,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/4c9b748f11d541cd806a5a0cd68b0003 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4c9b748f11d541cd806a5a0cd68b0003 2023-03-31 18:56:11,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:11,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:11,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:11,168 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:11,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:11,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:11,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:11,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,168 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4c9b748f11d541cd806a5a0cd68b0003, entries=150, sequenceid=698, filesize=11.9 K 2023-03-31 18:56:11,170 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:11,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/da93c37f9eb24dd3a0b794c4fb3b28a5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/da93c37f9eb24dd3a0b794c4fb3b28a5 2023-03-31 18:56:11,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:11,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:11,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,171 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:11,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:11,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:11,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:11,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:11,172 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:11,172 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:11,172 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:11,172 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:11,174 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/da93c37f9eb24dd3a0b794c4fb3b28a5, entries=150, sequenceid=698, filesize=11.9 K 2023-03-31 18:56:11,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:11,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,175 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 91eb2b257640c6018ad8eeccf0deb2cf in 856ms, sequenceid=698, compaction requested=true 2023-03-31 18:56:11,175 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,175 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:11,175 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:11,175 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:11,175 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:11,176 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:11,177 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35442 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:11,177 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:11,177 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:11,177 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/b33da7d897f74c409178c7a1c6e6d032, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/15a52e6c304147929dad28578a82ad52, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4c9b748f11d541cd806a5a0cd68b0003] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=34.6 K 2023-03-31 18:56:11,177 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35442 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:11,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,177 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:11,177 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:11,177 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43c9d0c41e714227a0f031f2aebb146d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/e3a7831b49ae43bfadc852ad3cf00fd6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/934b15ed2c2444af802fb18c018abecd] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=34.6 K 2023-03-31 18:56:11,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:11,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b92418[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:11,178 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:11,178 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,178 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting b33da7d897f74c409178c7a1c6e6d032, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1680288969162 2023-03-31 18:56:11,178 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 43c9d0c41e714227a0f031f2aebb146d, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1680288969162 2023-03-31 18:56:11,178 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 15a52e6c304147929dad28578a82ad52, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1680288969733 2023-03-31 18:56:11,178 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting e3a7831b49ae43bfadc852ad3cf00fd6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1680288969733 2023-03-31 18:56:11,179 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:11,179 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4c9b748f11d541cd806a5a0cd68b0003, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680288970311 2023-03-31 18:56:11,179 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 934b15ed2c2444af802fb18c018abecd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680288970311 2023-03-31 18:56:11,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:11,179 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:11,179 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,179 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:11,181 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:11,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:11,182 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:11,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:11,190 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:11,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:11,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:11,190 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:11,190 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:11,190 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#501 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:11,191 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#502 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:11,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:11,193 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,195 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:11,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:11,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:11,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:11,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:11,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:11,197 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:11,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:11,197 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:11,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:11,205 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/f5f69bbaefa546dc838aca48b14dc7d2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f5f69bbaefa546dc838aca48b14dc7d2 2023-03-31 18:56:11,211 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into f5f69bbaefa546dc838aca48b14dc7d2(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:11,211 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:11,211 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=13, startTime=1680288971175; duration=0sec 2023-03-31 18:56:11,212 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:11,212 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 18:56:11,212 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 18:56:11,212 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 18:56:11,213 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. because compaction request was cancelled 2023-03-31 18:56:11,214 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/0e8b08b8e8454e47a6523a0f1b751436 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0e8b08b8e8454e47a6523a0f1b751436 2023-03-31 18:56:11,219 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into 0e8b08b8e8454e47a6523a0f1b751436(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:11,220 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:11,220 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=13, startTime=1680288971175; duration=0sec 2023-03-31 18:56:11,220 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:11,298 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:11,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:11,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:11,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:11,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:11,301 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:11,301 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:11,301 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:11,301 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:11,301 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,305 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:11,305 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:11,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:11,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:11,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:11,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,307 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:11,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:11,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:11,307 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:11,307 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:11,307 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:11,307 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:11,307 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:11,307 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:11,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:11,308 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-31 18:56:11,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:11,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:11,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:11,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:11,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:11,308 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:11,322 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/011497b6835648af895ab1ce5e37d057 2023-03-31 18:56:11,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:11,558 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:11,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289031573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289031574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289031576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289031577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289031679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289031679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289031683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289031684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,733 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/4a67d682c5124210996354d5e3bd657d 2023-03-31 18:56:11,748 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/8dcf177a01a342ada611745fdd27f2b4 2023-03-31 18:56:11,751 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/011497b6835648af895ab1ce5e37d057 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/011497b6835648af895ab1ce5e37d057 2023-03-31 18:56:11,754 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/011497b6835648af895ab1ce5e37d057, entries=150, sequenceid=738, filesize=11.9 K 2023-03-31 18:56:11,755 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/4a67d682c5124210996354d5e3bd657d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4a67d682c5124210996354d5e3bd657d 2023-03-31 18:56:11,758 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4a67d682c5124210996354d5e3bd657d, entries=150, sequenceid=738, filesize=11.9 K 2023-03-31 18:56:11,759 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/8dcf177a01a342ada611745fdd27f2b4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8dcf177a01a342ada611745fdd27f2b4 2023-03-31 18:56:11,762 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8dcf177a01a342ada611745fdd27f2b4, entries=150, sequenceid=738, filesize=11.9 K 2023-03-31 18:56:11,763 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 91eb2b257640c6018ad8eeccf0deb2cf in 454ms, sequenceid=738, compaction requested=true 2023-03-31 18:56:11,763 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:11,763 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:11,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:11,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:11,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:11,763 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:11,765 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,765 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:11,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:11,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3901e86d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:11,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:11,767 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,768 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,769 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:11,769 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:11,769 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:11,769 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,770 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:11,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:11,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:11,770 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,770 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:11,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:11,772 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:11,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:11,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:11,772 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:11,772 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:11,773 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,774 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,776 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:11,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:11,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:11,779 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:11,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:11,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,779 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:11,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:11,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:11,780 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:11,780 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:11,880 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:11,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:11,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:11,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:11,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:11,884 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:11,884 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:11,884 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:11,884 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:11,884 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,885 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,887 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:11,887 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:11,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:11,887 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:56:11,888 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:11,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:11,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:11,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:11,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:11,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:11,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:11,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:11,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:11,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:11,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:11,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:11,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:11,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:11,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:11,890 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:11,890 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:11,890 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:11,890 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:11,890 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:11,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:11,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:11,891 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:11,892 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed93f91[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:11,894 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:11,894 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,896 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:11,896 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,896 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:11,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=750 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/1a613bffe3714c41bd809cd1c62a617b 2023-03-31 18:56:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:11,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:11,903 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:11,903 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:11,903 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:11,903 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:11,903 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:11,906 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,906 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,906 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:11,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,907 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:11,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:11,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,911 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:11,911 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,911 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,911 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:11,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:11,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:11,911 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:11,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:11,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:11,911 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:11,911 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,911 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,911 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:11,911 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:11,911 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:11,911 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:11,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:11,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:11,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289031924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289031926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:11,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289031927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:11,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289031928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,011 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:12,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:12,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:12,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:12,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:12,014 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:12,014 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:12,015 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:12,015 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:12,015 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,015 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,016 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:12,016 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:12,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:12,017 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,017 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:12,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:12,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:12,018 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,018 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:12,018 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:12,018 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:12,021 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,021 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,021 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,021 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f1b0351[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:12,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:12,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,023 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:12,024 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,024 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:12,025 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:12,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:12,027 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:12,027 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:12,027 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:12,027 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:12,027 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:12,029 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,029 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,029 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:12,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,029 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,030 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289032031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:12,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:12,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:12,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:12,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:12,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:12,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,036 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,035 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:12,036 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:12,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:12,036 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,036 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,036 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,036 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:12,036 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:12,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289032034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289032034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289032035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,136 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:12,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:12,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:12,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:12,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:12,141 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:12,141 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:12,141 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:12,141 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:12,142 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,142 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,143 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,143 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:12,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:12,143 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,143 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:12,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:12,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:12,145 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,145 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:12,145 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:12,145 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:12,146 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,146 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:12,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd674e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:12,148 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:12,148 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,153 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:12,153 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,153 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:12,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:12,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:12,154 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:12,155 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:12,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:12,157 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:12,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:12,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:12,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:12,157 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:12,158 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,158 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,158 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:12,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:12,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:12,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:12,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:12,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:12,163 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:12,163 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:12,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:12,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:12,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:12,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289032233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289032242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289032243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289032244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,263 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:12,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:12,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:12,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:12,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:12,266 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:12,266 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:12,266 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:12,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:12,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,267 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,269 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,270 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:12,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:12,270 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,270 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:12,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:12,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:12,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:12,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,271 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:12,272 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:12,272 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:12,273 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,273 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,274 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,274 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3639b5ba[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:12,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:12,275 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,277 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,277 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,277 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:12,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:12,279 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:12,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:12,282 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:12,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:12,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:12,282 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:12,282 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:12,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:12,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,285 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,288 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:12,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:12,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:12,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:12,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:12,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:12,290 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:12,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:12,290 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:12,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,291 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:12,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=750 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/decd493bbe0341d0a13afcd01b41888c 2023-03-31 18:56:12,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=750 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/e0b1bce130d64f9fa02eff540084f337 2023-03-31 18:56:12,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/1a613bffe3714c41bd809cd1c62a617b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1a613bffe3714c41bd809cd1c62a617b 2023-03-31 18:56:12,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1a613bffe3714c41bd809cd1c62a617b, entries=200, sequenceid=750, filesize=14.3 K 2023-03-31 18:56:12,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/decd493bbe0341d0a13afcd01b41888c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/decd493bbe0341d0a13afcd01b41888c 2023-03-31 18:56:12,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/decd493bbe0341d0a13afcd01b41888c, entries=150, sequenceid=750, filesize=11.9 K 2023-03-31 18:56:12,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/e0b1bce130d64f9fa02eff540084f337 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e0b1bce130d64f9fa02eff540084f337 2023-03-31 18:56:12,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e0b1bce130d64f9fa02eff540084f337, entries=150, sequenceid=750, filesize=11.9 K 2023-03-31 18:56:12,347 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 91eb2b257640c6018ad8eeccf0deb2cf in 460ms, sequenceid=750, compaction requested=true 2023-03-31 18:56:12,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:12,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:12,347 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:12,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:12,347 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:12,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:12,349 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:12,349 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40428 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:12,349 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:12,349 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:12,349 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,349 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,349 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f5f69bbaefa546dc838aca48b14dc7d2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4a67d682c5124210996354d5e3bd657d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/decd493bbe0341d0a13afcd01b41888c] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=37.1 K 2023-03-31 18:56:12,349 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0e8b08b8e8454e47a6523a0f1b751436, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/011497b6835648af895ab1ce5e37d057, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1a613bffe3714c41bd809cd1c62a617b] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=39.5 K 2023-03-31 18:56:12,349 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f5f69bbaefa546dc838aca48b14dc7d2, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680288970311 2023-03-31 18:56:12,349 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 0e8b08b8e8454e47a6523a0f1b751436, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680288970311 2023-03-31 18:56:12,350 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4a67d682c5124210996354d5e3bd657d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1680288970420 2023-03-31 18:56:12,350 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 011497b6835648af895ab1ce5e37d057, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1680288970420 2023-03-31 18:56:12,350 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting decd493bbe0341d0a13afcd01b41888c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680288971575 2023-03-31 18:56:12,350 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 1a613bffe3714c41bd809cd1c62a617b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680288971572 2023-03-31 18:56:12,355 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#509 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:12,358 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#510 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:12,378 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/73b9ded7cd2c46a7994fa674a5864b41 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/73b9ded7cd2c46a7994fa674a5864b41 2023-03-31 18:56:12,382 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into 73b9ded7cd2c46a7994fa674a5864b41(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:12,382 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:12,382 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=13, startTime=1680288972347; duration=0sec 2023-03-31 18:56:12,382 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:12,382 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:12,383 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50106 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:12,383 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:12,383 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,383 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/2c0f06e25eaa4c9d98bc649421ad8c77, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/da93c37f9eb24dd3a0b794c4fb3b28a5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8dcf177a01a342ada611745fdd27f2b4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e0b1bce130d64f9fa02eff540084f337] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=48.9 K 2023-03-31 18:56:12,384 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 2c0f06e25eaa4c9d98bc649421ad8c77, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1680288969700 2023-03-31 18:56:12,384 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting da93c37f9eb24dd3a0b794c4fb3b28a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680288970311 2023-03-31 18:56:12,384 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8dcf177a01a342ada611745fdd27f2b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1680288970420 2023-03-31 18:56:12,384 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting e0b1bce130d64f9fa02eff540084f337, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680288971575 2023-03-31 18:56:12,389 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#511 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:12,391 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:12,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:12,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:12,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:12,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:12,398 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:12,398 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:12,398 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:12,398 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:12,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,399 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,400 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,400 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:12,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:12,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:12,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:12,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:12,404 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,404 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:12,404 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:56:12,404 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:12,404 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:12,405 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:12,405 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:12,405 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:12,405 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:12,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:12,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:12,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289032555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289032558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289032559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289032563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289032664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289032666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289032669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289032669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,770 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/8a538648387741d1879fcd9c70648791 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/8a538648387741d1879fcd9c70648791 2023-03-31 18:56:12,773 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into 8a538648387741d1879fcd9c70648791(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:12,773 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:12,774 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=13, startTime=1680288972347; duration=0sec 2023-03-31 18:56:12,774 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:12,805 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/9dd7dfcbd45b43eb834548cc2aed3544 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9dd7dfcbd45b43eb834548cc2aed3544 2023-03-31 18:56:12,809 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into 9dd7dfcbd45b43eb834548cc2aed3544(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:12,809 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:12,809 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=12, startTime=1680288972347; duration=0sec 2023-03-31 18:56:12,809 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:12,817 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/de851cf7fe9343a891f2f478125af1e4 2023-03-31 18:56:12,826 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/9f108bfe9cc34107bdf9cdebc502b9de 2023-03-31 18:56:12,834 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/d5e22dbfdd204c7697f483e4e84f5c1a 2023-03-31 18:56:12,837 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/de851cf7fe9343a891f2f478125af1e4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/de851cf7fe9343a891f2f478125af1e4 2023-03-31 18:56:12,840 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/de851cf7fe9343a891f2f478125af1e4, entries=150, sequenceid=771, filesize=11.9 K 2023-03-31 18:56:12,841 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/9f108bfe9cc34107bdf9cdebc502b9de as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f108bfe9cc34107bdf9cdebc502b9de 2023-03-31 18:56:12,845 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f108bfe9cc34107bdf9cdebc502b9de, entries=150, sequenceid=771, filesize=11.9 K 2023-03-31 18:56:12,846 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/d5e22dbfdd204c7697f483e4e84f5c1a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d5e22dbfdd204c7697f483e4e84f5c1a 2023-03-31 18:56:12,853 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d5e22dbfdd204c7697f483e4e84f5c1a, entries=150, sequenceid=771, filesize=11.9 K 2023-03-31 18:56:12,853 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 91eb2b257640c6018ad8eeccf0deb2cf in 449ms, sequenceid=771, compaction requested=false 2023-03-31 18:56:12,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:12,854 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,854 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:12,854 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:12,854 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,854 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:12,854 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:12,857 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:12,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715b215b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:12,858 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:12,858 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,860 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,860 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:12,861 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,861 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:12,862 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:12,865 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:12,865 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:12,865 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:12,865 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:12,865 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:12,866 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,867 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,869 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,869 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,869 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:12,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,872 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,872 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:12,872 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:12,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:12,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:12,872 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:12,872 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:12,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:12,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:12,872 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,872 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,872 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,872 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,872 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,872 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:12,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:12,872 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:12,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:12,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 18:56:12,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:12,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:12,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:12,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:12,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:12,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:12,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/0df113fc42ae4324bb17991d7bfc5726 2023-03-31 18:56:12,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289032906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289032907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289032908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:12,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289032909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,973 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:12,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:12,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:12,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:12,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:12,976 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:12,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:12,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:12,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:12,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:12,978 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:12,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:12,979 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,979 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:12,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:12,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:12,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:12,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:12,980 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,980 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:12,980 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:12,980 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,981 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:12,981 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:12,982 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,982 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,982 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,982 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:12,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:12,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@664da5aa[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:12,985 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:12,985 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,986 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:12,986 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:12,986 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:12,986 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,986 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:12,986 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:12,991 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:12,991 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:12,991 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,991 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:12,993 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:12,994 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:12,994 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:12,994 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:12,994 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:12,995 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,995 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,995 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:12,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:12,997 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:12,999 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,001 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,001 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,001 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,001 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,001 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:13,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:13,001 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,001 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,001 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,001 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,001 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,001 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:13,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,001 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289033015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289033015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289033015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289033015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,101 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:13,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:13,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:13,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:13,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:13,105 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:13,105 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:13,105 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:13,105 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:13,105 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,105 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,107 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,107 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,107 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,107 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:13,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:13,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:13,108 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:13,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:13,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:13,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:13,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:13,109 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,109 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:13,109 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:13,109 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:13,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:13,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:13,109 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,115 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d51493a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:13,117 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:13,117 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,118 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:13,119 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:13,119 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:13,119 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,119 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,120 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:13,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:13,123 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:13,123 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:13,123 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:13,123 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:13,123 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:13,124 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,125 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,125 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,125 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,126 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,131 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,131 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,131 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,131 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,131 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,131 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,131 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:13,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:13,131 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,131 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,131 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,131 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:13,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,132 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:13,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289033222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289033222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289033223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289033224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,232 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:13,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:13,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:13,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:13,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:13,235 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:13,235 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:13,235 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:13,235 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:13,236 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,236 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,237 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,237 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,237 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:13,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:13,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:13,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:13,238 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:13,238 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:13,239 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,239 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:13,239 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:13,239 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:13,241 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:13,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:13,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763ea94f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:13,243 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:13,243 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,244 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,244 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:13,244 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:13,244 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:13,245 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,245 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:13,247 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,247 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:13,250 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:13,250 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:13,250 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:13,250 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:13,250 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:13,252 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,252 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,252 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,253 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,254 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,257 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,257 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,257 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,257 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,257 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,257 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,257 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,258 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,258 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:13,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:13,258 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,258 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:13,258 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:13,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/3ddac2a387fa469485c552a978e81d5f 2023-03-31 18:56:13,359 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:13,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:13,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:13,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:13,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:13,362 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:13,362 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:13,362 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:13,362 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:13,363 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,363 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,364 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,364 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,364 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,364 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:13,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:13,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:13,365 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:13,365 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:13,366 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,366 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:13,366 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:13,366 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:13,369 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ec70f2a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:13,371 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:13,371 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,372 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:13,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:13,372 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:13,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,372 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:13,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:13,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,375 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,375 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:13,377 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:13,378 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:13,378 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:13,378 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:13,378 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:13,380 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,380 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,380 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,381 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,383 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,388 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:13,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:13,388 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,389 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,389 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:13,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,389 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:13,489 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:13,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:13,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:13,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:13,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:13,492 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:13,492 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:13,492 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:13,492 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:13,493 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,493 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,494 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,494 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:13,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:13,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:13,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:13,495 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:13,495 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:13,495 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:13,495 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:13,495 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:13,495 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,495 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:13,495 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:13,495 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,496 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:13,496 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:13,496 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,496 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:13,496 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:13,497 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,497 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:13,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55ac81ad[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:13,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:13,499 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,500 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:13,500 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,500 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:13,502 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:13,504 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:13,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:13,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:13,504 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:13,505 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:13,506 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,506 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,506 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,507 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,508 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,510 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,511 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:13,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:13,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,511 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,511 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:13,511 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:13,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33156 deadline: 1680289033528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33136 deadline: 1680289033529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33188 deadline: 1680289033530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:13,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33150 deadline: 1680289033530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,611 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:13,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:13,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:13,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:13,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:13,615 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:13,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:13,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:13,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:13,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,615 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,616 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,616 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,617 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:13,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:13,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:13,618 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,618 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. started... 2023-03-31 18:56:13,618 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. as already flushing 2023-03-31 18:56:13,618 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:13,621 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b31abb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:13,622 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:13,622 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,623 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:13,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,624 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:13,626 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:13,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:13,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:13,628 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:13,628 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:13,628 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:13,628 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:13,628 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:13,630 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,630 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:13,630 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,631 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:13,631 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:13,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:13,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:13,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:13,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:13,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:13,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:13,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:13,636 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:13,636 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:13,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:13,636 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:13,636 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:13,636 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:13,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:13,636 DEBUG [Thread-3013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:13,636 DEBUG [Thread-3013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:13,712 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/58bc937aff224889bf301bd050771315 2023-03-31 18:56:13,715 DEBUG [Thread-3020] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x211582f2 to 127.0.0.1:63858 2023-03-31 18:56:13,715 DEBUG [Thread-3020] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:13,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/0df113fc42ae4324bb17991d7bfc5726 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0df113fc42ae4324bb17991d7bfc5726 2023-03-31 18:56:13,717 DEBUG [Thread-3016] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:63858 2023-03-31 18:56:13,717 DEBUG [Thread-3016] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:13,717 DEBUG [Thread-3014] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:63858 2023-03-31 18:56:13,718 DEBUG [Thread-3014] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:13,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0df113fc42ae4324bb17991d7bfc5726, entries=250, sequenceid=787, filesize=16.7 K 2023-03-31 18:56:13,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/3ddac2a387fa469485c552a978e81d5f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3ddac2a387fa469485c552a978e81d5f 2023-03-31 18:56:13,721 DEBUG [Thread-3018] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f5363d8 to 127.0.0.1:63858 2023-03-31 18:56:13,721 DEBUG [Thread-3018] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:13,721 DEBUG [Thread-3022] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65816b6e to 127.0.0.1:63858 2023-03-31 18:56:13,722 DEBUG [Thread-3022] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:13,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3ddac2a387fa469485c552a978e81d5f, entries=150, sequenceid=787, filesize=11.9 K 2023-03-31 18:56:13,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/58bc937aff224889bf301bd050771315 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/58bc937aff224889bf301bd050771315 2023-03-31 18:56:13,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/58bc937aff224889bf301bd050771315, entries=150, sequenceid=787, filesize=11.9 K 2023-03-31 18:56:13,728 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 91eb2b257640c6018ad8eeccf0deb2cf in 851ms, sequenceid=787, compaction requested=true 2023-03-31 18:56:13,728 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:13,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:13,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:13,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:13,728 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:13,728 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:13,729 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42970 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:13,729 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/A is initiating minor compaction (all files) 2023-03-31 18:56:13,729 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38090 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:13,729 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/A in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,729 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/B is initiating minor compaction (all files) 2023-03-31 18:56:13,729 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/73b9ded7cd2c46a7994fa674a5864b41, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/de851cf7fe9343a891f2f478125af1e4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0df113fc42ae4324bb17991d7bfc5726] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=42.0 K 2023-03-31 18:56:13,730 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/B in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,730 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/8a538648387741d1879fcd9c70648791, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f108bfe9cc34107bdf9cdebc502b9de, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3ddac2a387fa469485c552a978e81d5f] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=37.2 K 2023-03-31 18:56:13,730 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 73b9ded7cd2c46a7994fa674a5864b41, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680288971575 2023-03-31 18:56:13,730 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 8a538648387741d1879fcd9c70648791, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680288971575 2023-03-31 18:56:13,730 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting de851cf7fe9343a891f2f478125af1e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1680288971915 2023-03-31 18:56:13,730 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9f108bfe9cc34107bdf9cdebc502b9de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1680288971915 2023-03-31 18:56:13,730 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 0df113fc42ae4324bb17991d7bfc5726, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1680288972554 2023-03-31 18:56:13,730 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3ddac2a387fa469485c552a978e81d5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1680288972562 2023-03-31 18:56:13,735 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#B#compaction#519 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:13,735 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#A#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:13,736 DEBUG [Thread-3013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:13,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:13,742 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/a88a24a1b5464e9e80780d221527f430 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a88a24a1b5464e9e80780d221527f430 2023-03-31 18:56:13,745 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/B of 91eb2b257640c6018ad8eeccf0deb2cf into a88a24a1b5464e9e80780d221527f430(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:13,745 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:13,745 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/B, priority=13, startTime=1680288973728; duration=0sec 2023-03-31 18:56:13,745 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:13,746 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:13,746 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:13,746 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): 91eb2b257640c6018ad8eeccf0deb2cf/C is initiating minor compaction (all files) 2023-03-31 18:56:13,747 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 91eb2b257640c6018ad8eeccf0deb2cf/C in TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:13,747 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9dd7dfcbd45b43eb834548cc2aed3544, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d5e22dbfdd204c7697f483e4e84f5c1a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/58bc937aff224889bf301bd050771315] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp, totalSize=37.2 K 2023-03-31 18:56:13,747 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9dd7dfcbd45b43eb834548cc2aed3544, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680288971575 2023-03-31 18:56:13,747 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d5e22dbfdd204c7697f483e4e84f5c1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1680288971915 2023-03-31 18:56:13,747 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 58bc937aff224889bf301bd050771315, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1680288972562 2023-03-31 18:56:13,752 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): 91eb2b257640c6018ad8eeccf0deb2cf#C#compaction#520 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:13,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:13,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-31 18:56:13,875 DEBUG [Thread-3003] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x501c5a88 to 127.0.0.1:63858 2023-03-31 18:56:13,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:13,875 DEBUG [Thread-3003] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:13,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:13,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:13,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:13,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:13,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:14,036 DEBUG [Thread-3011] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06edd017 to 127.0.0.1:63858 2023-03-31 18:56:14,036 DEBUG [Thread-3011] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:14,037 DEBUG [Thread-3007] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:63858 2023-03-31 18:56:14,037 DEBUG [Thread-3007] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:14,038 DEBUG [Thread-3005] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27f2d997 to 127.0.0.1:63858 2023-03-31 18:56:14,038 DEBUG [Thread-3005] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:14,038 DEBUG [Thread-3009] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:63858 2023-03-31 18:56:14,038 DEBUG [Thread-3009] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:14,038 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 18:56:14,038 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 18 2023-03-31 18:56:14,038 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 155 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 148 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 154 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 187 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1672 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5015 rows 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1679 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5037 rows 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1681 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5043 rows 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1682 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5045 rows 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(402): scanned 1692 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(403): verified 5075 rows 2023-03-31 18:56:14,039 INFO [Listener at localhost/43327] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 18:56:14,039 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e372785 to 127.0.0.1:63858 2023-03-31 18:56:14,039 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:14,041 INFO [Listener at localhost/43327] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 18:56:14,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 18:56:14,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:14,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 18:56:14,045 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288974045"}]},"ts":"1680288974045"} 2023-03-31 18:56:14,046 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 18:56:14,143 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 18:56:14,144 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=91eb2b257640c6018ad8eeccf0deb2cf, UNASSIGN}] 2023-03-31 18:56:14,144 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/a79109603bb746529b3ea5b5d30a9b84 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/a79109603bb746529b3ea5b5d30a9b84 2023-03-31 18:56:14,145 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=91eb2b257640c6018ad8eeccf0deb2cf, UNASSIGN 2023-03-31 18:56:14,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 18:56:14,146 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=91eb2b257640c6018ad8eeccf0deb2cf, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:14,146 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288974146"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288974146"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288974146"}]},"ts":"1680288974146"} 2023-03-31 18:56:14,147 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:56:14,149 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/A of 91eb2b257640c6018ad8eeccf0deb2cf into a79109603bb746529b3ea5b5d30a9b84(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:14,149 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:14,149 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/A, priority=13, startTime=1680288973728; duration=0sec 2023-03-31 18:56:14,149 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:14,159 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/c32e9ccd07854a1980bc13846bad54c4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/c32e9ccd07854a1980bc13846bad54c4 2023-03-31 18:56:14,162 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 91eb2b257640c6018ad8eeccf0deb2cf/C of 91eb2b257640c6018ad8eeccf0deb2cf into c32e9ccd07854a1980bc13846bad54c4(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:14,163 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:14,163 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf., storeName=91eb2b257640c6018ad8eeccf0deb2cf/C, priority=13, startTime=1680288973728; duration=0sec 2023-03-31 18:56:14,163 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:14,281 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=807 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/57deada2be524340bff901fbad6b5909 2023-03-31 18:56:14,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:14,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 91eb2b257640c6018ad8eeccf0deb2cf, disabling compactions & flushes 2023-03-31 18:56:14,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:14,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 18:56:14,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 18:56:14,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=807 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/629ef7a9b38d46b39c02b6a4974644fd 2023-03-31 18:56:15,099 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=807 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/444dfaa8a2964750b34bb195774efed1 2023-03-31 18:56:15,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/57deada2be524340bff901fbad6b5909 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/57deada2be524340bff901fbad6b5909 2023-03-31 18:56:15,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/57deada2be524340bff901fbad6b5909, entries=150, sequenceid=807, filesize=11.9 K 2023-03-31 18:56:15,105 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/629ef7a9b38d46b39c02b6a4974644fd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/629ef7a9b38d46b39c02b6a4974644fd 2023-03-31 18:56:15,108 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/629ef7a9b38d46b39c02b6a4974644fd, entries=150, sequenceid=807, filesize=11.9 K 2023-03-31 18:56:15,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/444dfaa8a2964750b34bb195774efed1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/444dfaa8a2964750b34bb195774efed1 2023-03-31 18:56:15,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/444dfaa8a2964750b34bb195774efed1, entries=150, sequenceid=807, filesize=11.9 K 2023-03-31 18:56:15,112 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=26.84 KB/27480 for 91eb2b257640c6018ad8eeccf0deb2cf in 1237ms, sequenceid=807, compaction requested=false 2023-03-31 18:56:15,112 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:15,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:15,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:15,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. after waiting 0 ms 2023-03-31 18:56:15,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:15,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 91eb2b257640c6018ad8eeccf0deb2cf 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 18:56:15,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=A 2023-03-31 18:56:15,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:15,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=B 2023-03-31 18:56:15,113 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:15,113 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 91eb2b257640c6018ad8eeccf0deb2cf, store=C 2023-03-31 18:56:15,113 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:15,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 18:56:15,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=816 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/8ab26607cd6d4036ad5e5c31c4b4445e 2023-03-31 18:56:15,926 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=816 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/7b0868b5fdbd412cb48ac7979a7aa56d 2023-03-31 18:56:16,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 18:56:16,335 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=816 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/964ea8333c124d769320b5bf25d8cf61 2023-03-31 18:56:16,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/A/8ab26607cd6d4036ad5e5c31c4b4445e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/8ab26607cd6d4036ad5e5c31c4b4445e 2023-03-31 18:56:16,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/8ab26607cd6d4036ad5e5c31c4b4445e, entries=50, sequenceid=816, filesize=7.1 K 2023-03-31 18:56:16,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/B/7b0868b5fdbd412cb48ac7979a7aa56d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/7b0868b5fdbd412cb48ac7979a7aa56d 2023-03-31 18:56:16,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/7b0868b5fdbd412cb48ac7979a7aa56d, entries=50, sequenceid=816, filesize=7.1 K 2023-03-31 18:56:16,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/.tmp/C/964ea8333c124d769320b5bf25d8cf61 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/964ea8333c124d769320b5bf25d8cf61 2023-03-31 18:56:16,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/964ea8333c124d769320b5bf25d8cf61, entries=50, sequenceid=816, filesize=7.1 K 2023-03-31 18:56:16,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 91eb2b257640c6018ad8eeccf0deb2cf in 1235ms, sequenceid=816, compaction requested=true 2023-03-31 18:56:16,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/204d1c834d9a4f658d15e1329a46f5a0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/97f40eadc1734a98b12eeb27fd0992d1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b060a41d60984f12a4a7238eccc6f52d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78ac82fc77564700b84196ecbd6d10db, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/fdc0f42da5904a5a9cf9635421016a10, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/54e25367321a458aa345bb8f9f972e14, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/8927416edac04274b13e0e48ea00245d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/eda9b9c36833480aa4585d37450dee46, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c08fcabd6af74ba48c96a10d62e6e202, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43074f0c5e6a4b80a820f2e2678111aa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/42c605b5b2c04e8cacab6ed05e5edbeb, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/638325e7ebf84ed392d998501960464f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/24f28baaa40a45a79a15a3eb889c0430, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b0839aeac64249b6ab47be8994472604, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/7aa9d512aee346188c5d835a9f60b96e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40709ac3a3b64458b4a57e345d9b97c9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ddee0ab6ca5d4613b4d7d8fdc436c5e5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/a5952a20c13f4deeb147103bfe24178a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d9c31b315bea4a8bbc1f0ad09875c06b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/685143671c14496db0e3994ba5d79cf7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c1e9954d47bb4a098d8458f4d8bf83f2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/47ee6c1ca319478c8bc49863a6bf2577, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/403fd6a8570848debc1d3997b4334b17, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5acb6d4c7a6c48d2a03e5e992919481d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/bea9dda17bd54f798dd7db20f84d24aa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c372dafd1ef44c229e477a89d06db22a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ec20f74c0fdc47769b1310fad8c5d601, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/3c7174f1baac4c8a890d8a4a8828c26c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c783ced293654173b020cccf6050c782, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0f7bf7d6aeb04cad8152ccbdbd5e8745, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/6db2f1caeb4c402f8831732fede1ed23, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0070f1d2e30b48a4bcb9427e2e8e7f3f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1d10325cc3c04276a9b585af97fe324b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/74fd72c1edc1409db372e1452947de72, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d1d2dc836d794ce7bb08305cad6839f8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78f36b676735420e904bc0e589901276, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5afa97edeaaf4e93b052b9cccfc1692b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/158a52b24c2946cc8f0808004ad4c7cf, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40a043b6c2644e4a9662a9abd37b5b23, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43c9d0c41e714227a0f031f2aebb146d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/e3a7831b49ae43bfadc852ad3cf00fd6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0e8b08b8e8454e47a6523a0f1b751436, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/934b15ed2c2444af802fb18c018abecd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/011497b6835648af895ab1ce5e37d057, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1a613bffe3714c41bd809cd1c62a617b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/73b9ded7cd2c46a7994fa674a5864b41, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/de851cf7fe9343a891f2f478125af1e4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0df113fc42ae4324bb17991d7bfc5726] to archive 2023-03-31 18:56:16,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:56:16,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/204d1c834d9a4f658d15e1329a46f5a0 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/204d1c834d9a4f658d15e1329a46f5a0 2023-03-31 18:56:16,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/97f40eadc1734a98b12eeb27fd0992d1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/97f40eadc1734a98b12eeb27fd0992d1 2023-03-31 18:56:16,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b060a41d60984f12a4a7238eccc6f52d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b060a41d60984f12a4a7238eccc6f52d 2023-03-31 18:56:16,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78ac82fc77564700b84196ecbd6d10db to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78ac82fc77564700b84196ecbd6d10db 2023-03-31 18:56:16,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/fdc0f42da5904a5a9cf9635421016a10 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/fdc0f42da5904a5a9cf9635421016a10 2023-03-31 18:56:16,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/54e25367321a458aa345bb8f9f972e14 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/54e25367321a458aa345bb8f9f972e14 2023-03-31 18:56:16,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/8927416edac04274b13e0e48ea00245d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/8927416edac04274b13e0e48ea00245d 2023-03-31 18:56:16,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/eda9b9c36833480aa4585d37450dee46 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/eda9b9c36833480aa4585d37450dee46 2023-03-31 18:56:16,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c08fcabd6af74ba48c96a10d62e6e202 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c08fcabd6af74ba48c96a10d62e6e202 2023-03-31 18:56:16,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43074f0c5e6a4b80a820f2e2678111aa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43074f0c5e6a4b80a820f2e2678111aa 2023-03-31 18:56:16,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/42c605b5b2c04e8cacab6ed05e5edbeb to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/42c605b5b2c04e8cacab6ed05e5edbeb 2023-03-31 18:56:16,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/638325e7ebf84ed392d998501960464f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/638325e7ebf84ed392d998501960464f 2023-03-31 18:56:16,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/24f28baaa40a45a79a15a3eb889c0430 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/24f28baaa40a45a79a15a3eb889c0430 2023-03-31 18:56:16,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b0839aeac64249b6ab47be8994472604 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/b0839aeac64249b6ab47be8994472604 2023-03-31 18:56:16,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/7aa9d512aee346188c5d835a9f60b96e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/7aa9d512aee346188c5d835a9f60b96e 2023-03-31 18:56:16,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40709ac3a3b64458b4a57e345d9b97c9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40709ac3a3b64458b4a57e345d9b97c9 2023-03-31 18:56:16,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ddee0ab6ca5d4613b4d7d8fdc436c5e5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ddee0ab6ca5d4613b4d7d8fdc436c5e5 2023-03-31 18:56:16,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/a5952a20c13f4deeb147103bfe24178a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/a5952a20c13f4deeb147103bfe24178a 2023-03-31 18:56:16,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d9c31b315bea4a8bbc1f0ad09875c06b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d9c31b315bea4a8bbc1f0ad09875c06b 2023-03-31 18:56:16,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/685143671c14496db0e3994ba5d79cf7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/685143671c14496db0e3994ba5d79cf7 2023-03-31 18:56:16,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c1e9954d47bb4a098d8458f4d8bf83f2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c1e9954d47bb4a098d8458f4d8bf83f2 2023-03-31 18:56:16,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/47ee6c1ca319478c8bc49863a6bf2577 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/47ee6c1ca319478c8bc49863a6bf2577 2023-03-31 18:56:16,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/403fd6a8570848debc1d3997b4334b17 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/403fd6a8570848debc1d3997b4334b17 2023-03-31 18:56:16,370 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5acb6d4c7a6c48d2a03e5e992919481d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5acb6d4c7a6c48d2a03e5e992919481d 2023-03-31 18:56:16,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/bea9dda17bd54f798dd7db20f84d24aa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/bea9dda17bd54f798dd7db20f84d24aa 2023-03-31 18:56:16,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c372dafd1ef44c229e477a89d06db22a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c372dafd1ef44c229e477a89d06db22a 2023-03-31 18:56:16,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ec20f74c0fdc47769b1310fad8c5d601 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/ec20f74c0fdc47769b1310fad8c5d601 2023-03-31 18:56:16,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/3c7174f1baac4c8a890d8a4a8828c26c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/3c7174f1baac4c8a890d8a4a8828c26c 2023-03-31 18:56:16,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c783ced293654173b020cccf6050c782 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/c783ced293654173b020cccf6050c782 2023-03-31 18:56:16,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0f7bf7d6aeb04cad8152ccbdbd5e8745 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0f7bf7d6aeb04cad8152ccbdbd5e8745 2023-03-31 18:56:16,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/6db2f1caeb4c402f8831732fede1ed23 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/6db2f1caeb4c402f8831732fede1ed23 2023-03-31 18:56:16,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0070f1d2e30b48a4bcb9427e2e8e7f3f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0070f1d2e30b48a4bcb9427e2e8e7f3f 2023-03-31 18:56:16,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1d10325cc3c04276a9b585af97fe324b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1d10325cc3c04276a9b585af97fe324b 2023-03-31 18:56:16,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/74fd72c1edc1409db372e1452947de72 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/74fd72c1edc1409db372e1452947de72 2023-03-31 18:56:16,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d1d2dc836d794ce7bb08305cad6839f8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/d1d2dc836d794ce7bb08305cad6839f8 2023-03-31 18:56:16,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78f36b676735420e904bc0e589901276 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/78f36b676735420e904bc0e589901276 2023-03-31 18:56:16,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5afa97edeaaf4e93b052b9cccfc1692b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/5afa97edeaaf4e93b052b9cccfc1692b 2023-03-31 18:56:16,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/158a52b24c2946cc8f0808004ad4c7cf to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/158a52b24c2946cc8f0808004ad4c7cf 2023-03-31 18:56:16,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40a043b6c2644e4a9662a9abd37b5b23 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/40a043b6c2644e4a9662a9abd37b5b23 2023-03-31 18:56:16,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43c9d0c41e714227a0f031f2aebb146d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/43c9d0c41e714227a0f031f2aebb146d 2023-03-31 18:56:16,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/e3a7831b49ae43bfadc852ad3cf00fd6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/e3a7831b49ae43bfadc852ad3cf00fd6 2023-03-31 18:56:16,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0e8b08b8e8454e47a6523a0f1b751436 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0e8b08b8e8454e47a6523a0f1b751436 2023-03-31 18:56:16,388 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/934b15ed2c2444af802fb18c018abecd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/934b15ed2c2444af802fb18c018abecd 2023-03-31 18:56:16,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/011497b6835648af895ab1ce5e37d057 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/011497b6835648af895ab1ce5e37d057 2023-03-31 18:56:16,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1a613bffe3714c41bd809cd1c62a617b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/1a613bffe3714c41bd809cd1c62a617b 2023-03-31 18:56:16,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/73b9ded7cd2c46a7994fa674a5864b41 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/73b9ded7cd2c46a7994fa674a5864b41 2023-03-31 18:56:16,392 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/de851cf7fe9343a891f2f478125af1e4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/de851cf7fe9343a891f2f478125af1e4 2023-03-31 18:56:16,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0df113fc42ae4324bb17991d7bfc5726 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/0df113fc42ae4324bb17991d7bfc5726 2023-03-31 18:56:16,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/be8723ddee5b47bd984e0952d9051f52, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/fbb9332e519e4b9991b50da21a417492, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ff688b39096f42b29272c21d07f5f191, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/d2c406938768440faf55dc73e3d5b975, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a7f00c5cb02243128aff795b1f127978, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3caf35ee3fe944e19d7dd4fa352d97f6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/5a2985b499414c33a0d47ff4ade3d047, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/c79a36079ab943c0a20657fe301886fc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/81ee9051b2b24aff814055c0d0b7371c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6146e039e3e84efda8f3b5b813222b71, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/edf2adaca8484ea2b8bfbac92fb94316, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6469673853d948b58af430776267c5e1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/02ac0bbce212478590d862c5994b330c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/818b93c267984bda86676f85db42f53b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/094e4149252841c9ad563996e58e7985, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03e21819bc1d4e5199076983ca884804, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2f97de4b7de24d61908ebf79146e1f6c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3695dd0d155f4aa5b523b3d8b7e8d681, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4262c4a6c47b4803ba39f90a61f76681, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f7f55a4b50184b46a422d6da77e6d269, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed08b6cde848413daa487d4351961d55, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2705966f8f8e4a2987a876e393eb64a5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/46ef460961b944899130f9831c99b816, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/7fc3e3d7d8fe4445908ecefc8c2e856c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a59ff037f4444573a027b13a5c95f086, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ffe9a6537c8e455dbd8aec6672c96b13, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/abae065691e348b6ad4bd1e5dc2e3850, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/bf1b44b092114f5ea721f47875a1da0e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/0fb3671dede0450ca10b80a85911e313, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/163b742d11c4485c93dfc0b0fefe8ac2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed0ed0a23de84ccf9cfee1a928bfbf11, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/56b34c314eca49cda2c1ed865329598c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/95fdc54a0baa4e6b852da08831e5c5b4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/63d7d14743854db9aaae8dbfb76ec10e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4e8b49f823004086a8e1020b1581086f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/92294d27f80246538923ff6bd43639c3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03accf55ca7e452dae5f2894b8c85ffd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f563e3ec6f74703938f632bbe62bce7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/b33da7d897f74c409178c7a1c6e6d032, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a66d8161738c42909a4e9e28d5b629f2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/15a52e6c304147929dad28578a82ad52, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f5f69bbaefa546dc838aca48b14dc7d2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4c9b748f11d541cd806a5a0cd68b0003, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4a67d682c5124210996354d5e3bd657d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/8a538648387741d1879fcd9c70648791, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/decd493bbe0341d0a13afcd01b41888c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f108bfe9cc34107bdf9cdebc502b9de, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3ddac2a387fa469485c552a978e81d5f] to archive 2023-03-31 18:56:16,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:56:16,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/be8723ddee5b47bd984e0952d9051f52 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/be8723ddee5b47bd984e0952d9051f52 2023-03-31 18:56:16,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/fbb9332e519e4b9991b50da21a417492 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/fbb9332e519e4b9991b50da21a417492 2023-03-31 18:56:16,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ff688b39096f42b29272c21d07f5f191 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ff688b39096f42b29272c21d07f5f191 2023-03-31 18:56:16,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/d2c406938768440faf55dc73e3d5b975 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/d2c406938768440faf55dc73e3d5b975 2023-03-31 18:56:16,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a7f00c5cb02243128aff795b1f127978 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a7f00c5cb02243128aff795b1f127978 2023-03-31 18:56:16,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3caf35ee3fe944e19d7dd4fa352d97f6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3caf35ee3fe944e19d7dd4fa352d97f6 2023-03-31 18:56:16,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/5a2985b499414c33a0d47ff4ade3d047 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/5a2985b499414c33a0d47ff4ade3d047 2023-03-31 18:56:16,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/c79a36079ab943c0a20657fe301886fc to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/c79a36079ab943c0a20657fe301886fc 2023-03-31 18:56:16,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/81ee9051b2b24aff814055c0d0b7371c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/81ee9051b2b24aff814055c0d0b7371c 2023-03-31 18:56:16,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6146e039e3e84efda8f3b5b813222b71 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6146e039e3e84efda8f3b5b813222b71 2023-03-31 18:56:16,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/edf2adaca8484ea2b8bfbac92fb94316 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/edf2adaca8484ea2b8bfbac92fb94316 2023-03-31 18:56:16,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6469673853d948b58af430776267c5e1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/6469673853d948b58af430776267c5e1 2023-03-31 18:56:16,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/02ac0bbce212478590d862c5994b330c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/02ac0bbce212478590d862c5994b330c 2023-03-31 18:56:16,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/818b93c267984bda86676f85db42f53b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/818b93c267984bda86676f85db42f53b 2023-03-31 18:56:16,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/094e4149252841c9ad563996e58e7985 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/094e4149252841c9ad563996e58e7985 2023-03-31 18:56:16,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03e21819bc1d4e5199076983ca884804 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03e21819bc1d4e5199076983ca884804 2023-03-31 18:56:16,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2f97de4b7de24d61908ebf79146e1f6c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2f97de4b7de24d61908ebf79146e1f6c 2023-03-31 18:56:16,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3695dd0d155f4aa5b523b3d8b7e8d681 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3695dd0d155f4aa5b523b3d8b7e8d681 2023-03-31 18:56:16,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4262c4a6c47b4803ba39f90a61f76681 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4262c4a6c47b4803ba39f90a61f76681 2023-03-31 18:56:16,414 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f7f55a4b50184b46a422d6da77e6d269 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f7f55a4b50184b46a422d6da77e6d269 2023-03-31 18:56:16,415 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed08b6cde848413daa487d4351961d55 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed08b6cde848413daa487d4351961d55 2023-03-31 18:56:16,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2705966f8f8e4a2987a876e393eb64a5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/2705966f8f8e4a2987a876e393eb64a5 2023-03-31 18:56:16,417 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/46ef460961b944899130f9831c99b816 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/46ef460961b944899130f9831c99b816 2023-03-31 18:56:16,417 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/7fc3e3d7d8fe4445908ecefc8c2e856c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/7fc3e3d7d8fe4445908ecefc8c2e856c 2023-03-31 18:56:16,418 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a59ff037f4444573a027b13a5c95f086 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a59ff037f4444573a027b13a5c95f086 2023-03-31 18:56:16,419 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ffe9a6537c8e455dbd8aec6672c96b13 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ffe9a6537c8e455dbd8aec6672c96b13 2023-03-31 18:56:16,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/abae065691e348b6ad4bd1e5dc2e3850 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/abae065691e348b6ad4bd1e5dc2e3850 2023-03-31 18:56:16,421 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/bf1b44b092114f5ea721f47875a1da0e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/bf1b44b092114f5ea721f47875a1da0e 2023-03-31 18:56:16,422 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/0fb3671dede0450ca10b80a85911e313 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/0fb3671dede0450ca10b80a85911e313 2023-03-31 18:56:16,423 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/163b742d11c4485c93dfc0b0fefe8ac2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/163b742d11c4485c93dfc0b0fefe8ac2 2023-03-31 18:56:16,424 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed0ed0a23de84ccf9cfee1a928bfbf11 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/ed0ed0a23de84ccf9cfee1a928bfbf11 2023-03-31 18:56:16,425 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/56b34c314eca49cda2c1ed865329598c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/56b34c314eca49cda2c1ed865329598c 2023-03-31 18:56:16,426 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/95fdc54a0baa4e6b852da08831e5c5b4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/95fdc54a0baa4e6b852da08831e5c5b4 2023-03-31 18:56:16,426 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/63d7d14743854db9aaae8dbfb76ec10e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/63d7d14743854db9aaae8dbfb76ec10e 2023-03-31 18:56:16,427 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4e8b49f823004086a8e1020b1581086f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4e8b49f823004086a8e1020b1581086f 2023-03-31 18:56:16,428 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/92294d27f80246538923ff6bd43639c3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/92294d27f80246538923ff6bd43639c3 2023-03-31 18:56:16,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03accf55ca7e452dae5f2894b8c85ffd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/03accf55ca7e452dae5f2894b8c85ffd 2023-03-31 18:56:16,431 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f563e3ec6f74703938f632bbe62bce7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f563e3ec6f74703938f632bbe62bce7 2023-03-31 18:56:16,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/b33da7d897f74c409178c7a1c6e6d032 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/b33da7d897f74c409178c7a1c6e6d032 2023-03-31 18:56:16,433 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a66d8161738c42909a4e9e28d5b629f2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a66d8161738c42909a4e9e28d5b629f2 2023-03-31 18:56:16,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/15a52e6c304147929dad28578a82ad52 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/15a52e6c304147929dad28578a82ad52 2023-03-31 18:56:16,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f5f69bbaefa546dc838aca48b14dc7d2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/f5f69bbaefa546dc838aca48b14dc7d2 2023-03-31 18:56:16,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4c9b748f11d541cd806a5a0cd68b0003 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4c9b748f11d541cd806a5a0cd68b0003 2023-03-31 18:56:16,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4a67d682c5124210996354d5e3bd657d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/4a67d682c5124210996354d5e3bd657d 2023-03-31 18:56:16,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/8a538648387741d1879fcd9c70648791 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/8a538648387741d1879fcd9c70648791 2023-03-31 18:56:16,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/decd493bbe0341d0a13afcd01b41888c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/decd493bbe0341d0a13afcd01b41888c 2023-03-31 18:56:16,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f108bfe9cc34107bdf9cdebc502b9de to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/9f108bfe9cc34107bdf9cdebc502b9de 2023-03-31 18:56:16,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3ddac2a387fa469485c552a978e81d5f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/3ddac2a387fa469485c552a978e81d5f 2023-03-31 18:56:16,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/97e004c07a9f4b55b7482bbc89a9fbfa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/791bbf757f524087aa22e0f99b5371d8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/18b71d45c85d4e228b67f742abf7fafd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/b793c6d321494322b4373035069f6ac1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d0014c81eb9943deaf5d2433b997b92b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/53df9b15bee346de8e074120585b7ad7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/31c10f2381ee47f9bf680b9ad9a66789, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4ed04bd63553465b9f784d920706c788, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a671ec556fc4433f9ad5335ae7b3c980, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/77ee22d973474b5cb0cee4fa5489f35e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9731184289a545deb1041e0a45b8eb74, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e5672c8632694429977552eb95dea18c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/bf65d00c94724257ace347e611a156c1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3ca152cb754445ec8feaee90b9c771be, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/61b381d7a3a54a94a170a33797974bec, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/43bb42247d764e28a8831443263f0747, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/c36100c76af94f62a8ab691a3ec45e1d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/cb153303bbd949019c1c757bffa61a17, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/0085128a12e54914a84445d4b9b715ae, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/fdc183b8ba5e48e6af92edf77861baf3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/6efd9765e7184d938693031ebc22d077, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/636a672236a3438abf015cea5d6380ce, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/736855a50e184ffb8a251da5cf82b370, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/ecc0d9fefffd4a24b0a5f4de3b12bb32, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/15a1772f87dd4c19a5b616cd7df9b453, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5db43d7b83c24c2b8f06b3b478de635f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3d8fdecc991a4e13ab41914989f263c8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8fa9c204fc324974aba480c62a03bc97, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/0084edf3cdcb4eeab9d3559c1c07535f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/64a229e92d414f24811a8ca6ffc92467, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4e6f8141edad46e8903a9a5ad11ce77f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e3209507cbd44250a3caa4a565be8b9d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/7f7a251540e14da8ab9ff81f242aead5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/78fb767f482b4659af486913b9dbd579, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5244e0fba4b94047827db8e8feb6b2e5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d139a2bf30354f8396c380aa05c58f27, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/65f52070ab26437cabe83ad111401412, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/720fe359cf974a5da7aaf67c65c63138, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a460289a6f5a4028824d50a9acf09055, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/2c0f06e25eaa4c9d98bc649421ad8c77, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/f6201811b08a440fbfe413bbe1cb0ba3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/da93c37f9eb24dd3a0b794c4fb3b28a5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8dcf177a01a342ada611745fdd27f2b4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9dd7dfcbd45b43eb834548cc2aed3544, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e0b1bce130d64f9fa02eff540084f337, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d5e22dbfdd204c7697f483e4e84f5c1a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/58bc937aff224889bf301bd050771315] to archive 2023-03-31 18:56:16,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:56:16,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/97e004c07a9f4b55b7482bbc89a9fbfa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/97e004c07a9f4b55b7482bbc89a9fbfa 2023-03-31 18:56:16,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/791bbf757f524087aa22e0f99b5371d8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/791bbf757f524087aa22e0f99b5371d8 2023-03-31 18:56:16,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/18b71d45c85d4e228b67f742abf7fafd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/18b71d45c85d4e228b67f742abf7fafd 2023-03-31 18:56:16,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/b793c6d321494322b4373035069f6ac1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/b793c6d321494322b4373035069f6ac1 2023-03-31 18:56:16,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d0014c81eb9943deaf5d2433b997b92b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d0014c81eb9943deaf5d2433b997b92b 2023-03-31 18:56:16,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/53df9b15bee346de8e074120585b7ad7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/53df9b15bee346de8e074120585b7ad7 2023-03-31 18:56:16,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/31c10f2381ee47f9bf680b9ad9a66789 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/31c10f2381ee47f9bf680b9ad9a66789 2023-03-31 18:56:16,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4ed04bd63553465b9f784d920706c788 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4ed04bd63553465b9f784d920706c788 2023-03-31 18:56:16,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a671ec556fc4433f9ad5335ae7b3c980 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a671ec556fc4433f9ad5335ae7b3c980 2023-03-31 18:56:16,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/77ee22d973474b5cb0cee4fa5489f35e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/77ee22d973474b5cb0cee4fa5489f35e 2023-03-31 18:56:16,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9731184289a545deb1041e0a45b8eb74 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9731184289a545deb1041e0a45b8eb74 2023-03-31 18:56:16,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e5672c8632694429977552eb95dea18c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e5672c8632694429977552eb95dea18c 2023-03-31 18:56:16,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/bf65d00c94724257ace347e611a156c1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/bf65d00c94724257ace347e611a156c1 2023-03-31 18:56:16,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3ca152cb754445ec8feaee90b9c771be to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3ca152cb754445ec8feaee90b9c771be 2023-03-31 18:56:16,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/61b381d7a3a54a94a170a33797974bec to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/61b381d7a3a54a94a170a33797974bec 2023-03-31 18:56:16,459 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/43bb42247d764e28a8831443263f0747 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/43bb42247d764e28a8831443263f0747 2023-03-31 18:56:16,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/c36100c76af94f62a8ab691a3ec45e1d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/c36100c76af94f62a8ab691a3ec45e1d 2023-03-31 18:56:16,461 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/cb153303bbd949019c1c757bffa61a17 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/cb153303bbd949019c1c757bffa61a17 2023-03-31 18:56:16,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/0085128a12e54914a84445d4b9b715ae to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/0085128a12e54914a84445d4b9b715ae 2023-03-31 18:56:16,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/fdc183b8ba5e48e6af92edf77861baf3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/fdc183b8ba5e48e6af92edf77861baf3 2023-03-31 18:56:16,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/6efd9765e7184d938693031ebc22d077 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/6efd9765e7184d938693031ebc22d077 2023-03-31 18:56:16,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/636a672236a3438abf015cea5d6380ce to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/636a672236a3438abf015cea5d6380ce 2023-03-31 18:56:16,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/736855a50e184ffb8a251da5cf82b370 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/736855a50e184ffb8a251da5cf82b370 2023-03-31 18:56:16,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/ecc0d9fefffd4a24b0a5f4de3b12bb32 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/ecc0d9fefffd4a24b0a5f4de3b12bb32 2023-03-31 18:56:16,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/15a1772f87dd4c19a5b616cd7df9b453 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/15a1772f87dd4c19a5b616cd7df9b453 2023-03-31 18:56:16,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5db43d7b83c24c2b8f06b3b478de635f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5db43d7b83c24c2b8f06b3b478de635f 2023-03-31 18:56:16,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3d8fdecc991a4e13ab41914989f263c8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/3d8fdecc991a4e13ab41914989f263c8 2023-03-31 18:56:16,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8fa9c204fc324974aba480c62a03bc97 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8fa9c204fc324974aba480c62a03bc97 2023-03-31 18:56:16,470 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/0084edf3cdcb4eeab9d3559c1c07535f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/0084edf3cdcb4eeab9d3559c1c07535f 2023-03-31 18:56:16,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/64a229e92d414f24811a8ca6ffc92467 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/64a229e92d414f24811a8ca6ffc92467 2023-03-31 18:56:16,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4e6f8141edad46e8903a9a5ad11ce77f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/4e6f8141edad46e8903a9a5ad11ce77f 2023-03-31 18:56:16,473 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e3209507cbd44250a3caa4a565be8b9d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e3209507cbd44250a3caa4a565be8b9d 2023-03-31 18:56:16,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/7f7a251540e14da8ab9ff81f242aead5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/7f7a251540e14da8ab9ff81f242aead5 2023-03-31 18:56:16,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/78fb767f482b4659af486913b9dbd579 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/78fb767f482b4659af486913b9dbd579 2023-03-31 18:56:16,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5244e0fba4b94047827db8e8feb6b2e5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/5244e0fba4b94047827db8e8feb6b2e5 2023-03-31 18:56:16,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d139a2bf30354f8396c380aa05c58f27 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d139a2bf30354f8396c380aa05c58f27 2023-03-31 18:56:16,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/65f52070ab26437cabe83ad111401412 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/65f52070ab26437cabe83ad111401412 2023-03-31 18:56:16,478 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/720fe359cf974a5da7aaf67c65c63138 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/720fe359cf974a5da7aaf67c65c63138 2023-03-31 18:56:16,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a460289a6f5a4028824d50a9acf09055 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/a460289a6f5a4028824d50a9acf09055 2023-03-31 18:56:16,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/2c0f06e25eaa4c9d98bc649421ad8c77 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/2c0f06e25eaa4c9d98bc649421ad8c77 2023-03-31 18:56:16,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/f6201811b08a440fbfe413bbe1cb0ba3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/f6201811b08a440fbfe413bbe1cb0ba3 2023-03-31 18:56:16,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/da93c37f9eb24dd3a0b794c4fb3b28a5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/da93c37f9eb24dd3a0b794c4fb3b28a5 2023-03-31 18:56:16,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8dcf177a01a342ada611745fdd27f2b4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/8dcf177a01a342ada611745fdd27f2b4 2023-03-31 18:56:16,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9dd7dfcbd45b43eb834548cc2aed3544 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/9dd7dfcbd45b43eb834548cc2aed3544 2023-03-31 18:56:16,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e0b1bce130d64f9fa02eff540084f337 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/e0b1bce130d64f9fa02eff540084f337 2023-03-31 18:56:16,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d5e22dbfdd204c7697f483e4e84f5c1a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/d5e22dbfdd204c7697f483e4e84f5c1a 2023-03-31 18:56:16,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/58bc937aff224889bf301bd050771315 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/58bc937aff224889bf301bd050771315 2023-03-31 18:56:16,489 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/recovered.edits/819.seqid, newMaxSeqId=819, maxSeqId=1 2023-03-31 18:56:16,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf. 2023-03-31 18:56:16,491 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 91eb2b257640c6018ad8eeccf0deb2cf: 2023-03-31 18:56:16,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:16,493 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=91eb2b257640c6018ad8eeccf0deb2cf, regionState=CLOSED 2023-03-31 18:56:16,493 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288976493"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288976493"}]},"ts":"1680288976493"} 2023-03-31 18:56:16,495 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-31 18:56:16,495 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 91eb2b257640c6018ad8eeccf0deb2cf, server=jenkins-hbase4.apache.org,34951,1680288816339 in 2.3470 sec 2023-03-31 18:56:16,496 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-31 18:56:16,496 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=91eb2b257640c6018ad8eeccf0deb2cf, UNASSIGN in 2.3510 sec 2023-03-31 18:56:16,496 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288976496"}]},"ts":"1680288976496"} 2023-03-31 18:56:16,497 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 18:56:16,499 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 18:56:16,500 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.4580 sec 2023-03-31 18:56:18,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 18:56:18,151 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-31 18:56:18,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 18:56:18,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:18,154 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:18,154 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:18,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-31 18:56:18,159 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:18,160 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/recovered.edits] 2023-03-31 18:56:18,162 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/57deada2be524340bff901fbad6b5909 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/57deada2be524340bff901fbad6b5909 2023-03-31 18:56:18,163 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/8ab26607cd6d4036ad5e5c31c4b4445e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/8ab26607cd6d4036ad5e5c31c4b4445e 2023-03-31 18:56:18,165 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/a79109603bb746529b3ea5b5d30a9b84 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/A/a79109603bb746529b3ea5b5d30a9b84 2023-03-31 18:56:18,166 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/629ef7a9b38d46b39c02b6a4974644fd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/629ef7a9b38d46b39c02b6a4974644fd 2023-03-31 18:56:18,168 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/7b0868b5fdbd412cb48ac7979a7aa56d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/7b0868b5fdbd412cb48ac7979a7aa56d 2023-03-31 18:56:18,169 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a88a24a1b5464e9e80780d221527f430 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/B/a88a24a1b5464e9e80780d221527f430 2023-03-31 18:56:18,171 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/444dfaa8a2964750b34bb195774efed1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/444dfaa8a2964750b34bb195774efed1 2023-03-31 18:56:18,172 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/964ea8333c124d769320b5bf25d8cf61 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/964ea8333c124d769320b5bf25d8cf61 2023-03-31 18:56:18,173 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/c32e9ccd07854a1980bc13846bad54c4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/C/c32e9ccd07854a1980bc13846bad54c4 2023-03-31 18:56:18,175 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/recovered.edits/819.seqid to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf/recovered.edits/819.seqid 2023-03-31 18:56:18,176 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/91eb2b257640c6018ad8eeccf0deb2cf 2023-03-31 18:56:18,176 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:56:18,178 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:18,179 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 18:56:18,180 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 18:56:18,181 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:18,181 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 18:56:18,181 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680288978181"}]},"ts":"9223372036854775807"} 2023-03-31 18:56:18,183 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 18:56:18,183 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 91eb2b257640c6018ad8eeccf0deb2cf, NAME => 'TestAcidGuarantees,,1680288953050.91eb2b257640c6018ad8eeccf0deb2cf.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 18:56:18,183 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 18:56:18,183 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680288978183"}]},"ts":"9223372036854775807"} 2023-03-31 18:56:18,184 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 18:56:18,186 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:18,186 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 34 msec 2023-03-31 18:56:18,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-31 18:56:18,255 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-31 18:56:18,267 INFO [Listener at localhost/43327] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 226) - Thread LEAK? -, OpenFileDescriptor=591 (was 534) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=353 (was 296) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3762 (was 3419) - AvailableMemoryMB LEAK? - 2023-03-31 18:56:18,279 INFO [Listener at localhost/43327] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=591, MaxFileDescriptor=60000, SystemLoadAverage=353, ProcessCount=169, AvailableMemoryMB=3761 2023-03-31 18:56:18,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 18:56:18,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 18:56:18,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:18,283 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 18:56:18,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-31 18:56:18,283 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 18:56:18,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 18:56:18,284 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:18,285 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610 empty. 2023-03-31 18:56:18,285 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:18,285 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:56:18,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 18:56:18,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 18:56:18,694 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 18:56:18,696 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => f34b89af9bae1a93ef5b2acb275cb610, NAME => 'TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp 2023-03-31 18:56:18,702 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:56:18,702 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing f34b89af9bae1a93ef5b2acb275cb610, disabling compactions & flushes 2023-03-31 18:56:18,702 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:18,702 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:18,702 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. after waiting 0 ms 2023-03-31 18:56:18,702 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:18,702 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:18,702 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:18,704 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 18:56:18,705 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288978705"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288978705"}]},"ts":"1680288978705"} 2023-03-31 18:56:18,706 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 18:56:18,707 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 18:56:18,707 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288978707"}]},"ts":"1680288978707"} 2023-03-31 18:56:18,707 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 18:56:18,711 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, ASSIGN}] 2023-03-31 18:56:18,713 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, ASSIGN 2023-03-31 18:56:18,713 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34951,1680288816339; forceNewPlan=false, retain=false 2023-03-31 18:56:18,865 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=f34b89af9bae1a93ef5b2acb275cb610, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:18,865 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288978865"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288978865"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288978865"}]},"ts":"1680288978865"} 2023-03-31 18:56:18,866 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:56:18,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 18:56:19,022 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:19,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f34b89af9bae1a93ef5b2acb275cb610, NAME => 'TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:56:19,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:56:19,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,024 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,025 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A 2023-03-31 18:56:19,025 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A 2023-03-31 18:56:19,025 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:56:19,025 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f34b89af9bae1a93ef5b2acb275cb610 columnFamilyName A 2023-03-31 18:56:19,026 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(310): Store=f34b89af9bae1a93ef5b2acb275cb610/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:56:19,026 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,027 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B 2023-03-31 18:56:19,027 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B 2023-03-31 18:56:19,027 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:56:19,027 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f34b89af9bae1a93ef5b2acb275cb610 columnFamilyName B 2023-03-31 18:56:19,028 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(310): Store=f34b89af9bae1a93ef5b2acb275cb610/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:56:19,028 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,029 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C 2023-03-31 18:56:19,029 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C 2023-03-31 18:56:19,029 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:56:19,029 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f34b89af9bae1a93ef5b2acb275cb610 columnFamilyName C 2023-03-31 18:56:19,029 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(310): Store=f34b89af9bae1a93ef5b2acb275cb610/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:56:19,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:19,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:56:19,033 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 18:56:19,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f34b89af9bae1a93ef5b2acb275cb610; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61659465, jitterRate=-0.08120237290859222}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:56:19,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:19,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., pid=56, masterSystemTime=1680288979018 2023-03-31 18:56:19,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:19,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:19,038 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=f34b89af9bae1a93ef5b2acb275cb610, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:19,038 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288979038"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680288979038"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680288979038"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680288979038"}]},"ts":"1680288979038"} 2023-03-31 18:56:19,040 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-31 18:56:19,040 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 in 173 msec 2023-03-31 18:56:19,041 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-31 18:56:19,041 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, ASSIGN in 329 msec 2023-03-31 18:56:19,041 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 18:56:19,042 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680288979042"}]},"ts":"1680288979042"} 2023-03-31 18:56:19,042 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 18:56:19,047 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 18:56:19,048 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 766 msec 2023-03-31 18:56:19,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 18:56:19,387 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-31 18:56:19,388 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x3bafb41f to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:19,393 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68067de1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:19,394 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:19,396 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32986, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:19,396 DEBUG [Listener at localhost/43327] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 18:56:19,399 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59880, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 18:56:19,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 18:56:19,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 18:56:19,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:19,814 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 18:56:19,815 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 18:56:19,815 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 18:56:19,817 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-31 18:56:19,819 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, REOPEN/MOVE}] 2023-03-31 18:56:19,819 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, REOPEN/MOVE 2023-03-31 18:56:19,820 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=f34b89af9bae1a93ef5b2acb275cb610, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:19,820 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288979820"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288979820"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288979820"}]},"ts":"1680288979820"} 2023-03-31 18:56:19,821 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:56:19,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f34b89af9bae1a93ef5b2acb275cb610, disabling compactions & flushes 2023-03-31 18:56:19,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:19,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:19,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. after waiting 0 ms 2023-03-31 18:56:19,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:19,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-31 18:56:19,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:19,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:19,977 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: f34b89af9bae1a93ef5b2acb275cb610 to self. 2023-03-31 18:56:19,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:19,979 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=f34b89af9bae1a93ef5b2acb275cb610, regionState=CLOSED 2023-03-31 18:56:19,979 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288979979"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680288979979"}]},"ts":"1680288979979"} 2023-03-31 18:56:19,981 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-31 18:56:19,981 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 in 159 msec 2023-03-31 18:56:19,981 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34951,1680288816339; forceNewPlan=false, retain=true 2023-03-31 18:56:20,132 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=f34b89af9bae1a93ef5b2acb275cb610, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,132 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288980132"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680288980132"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680288980132"}]},"ts":"1680288980132"} 2023-03-31 18:56:20,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:56:20,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f34b89af9bae1a93ef5b2acb275cb610, NAME => 'TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.', STARTKEY => '', ENDKEY => ''} 2023-03-31 18:56:20,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:20,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 18:56:20,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:20,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:20,289 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:20,290 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A 2023-03-31 18:56:20,290 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A 2023-03-31 18:56:20,290 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:56:20,290 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f34b89af9bae1a93ef5b2acb275cb610 columnFamilyName A 2023-03-31 18:56:20,291 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(310): Store=f34b89af9bae1a93ef5b2acb275cb610/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:56:20,291 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:20,292 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B 2023-03-31 18:56:20,292 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B 2023-03-31 18:56:20,292 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:56:20,293 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f34b89af9bae1a93ef5b2acb275cb610 columnFamilyName B 2023-03-31 18:56:20,293 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(310): Store=f34b89af9bae1a93ef5b2acb275cb610/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:56:20,293 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:20,294 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C 2023-03-31 18:56:20,294 DEBUG [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C 2023-03-31 18:56:20,294 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 18:56:20,294 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f34b89af9bae1a93ef5b2acb275cb610 columnFamilyName C 2023-03-31 18:56:20,295 INFO [StoreOpener-f34b89af9bae1a93ef5b2acb275cb610-1] regionserver.HStore(310): Store=f34b89af9bae1a93ef5b2acb275cb610/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 18:56:20,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:20,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:20,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 18:56:20,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:20,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f34b89af9bae1a93ef5b2acb275cb610; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70101638, jitterRate=0.044595807790756226}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 18:56:20,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:20,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., pid=61, masterSystemTime=1680288980284 2023-03-31 18:56:20,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,302 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,302 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=f34b89af9bae1a93ef5b2acb275cb610, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,302 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680288980302"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680288980302"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680288980302"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680288980302"}]},"ts":"1680288980302"} 2023-03-31 18:56:20,304 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-31 18:56:20,304 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 in 170 msec 2023-03-31 18:56:20,305 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-31 18:56:20,305 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, REOPEN/MOVE in 485 msec 2023-03-31 18:56:20,306 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-31 18:56:20,307 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-03-31 18:56:20,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-03-31 18:56:20,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-31 18:56:20,309 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x03ced4ba to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:20,315 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45878790, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:20,315 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x63da10ca to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:20,319 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1644730f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:20,319 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x59185ba7 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:20,323 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af74c0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:20,323 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x062b3caa to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:20,326 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d759aab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:20,326 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x66b5b1f3 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:20,331 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39241fe3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:20,332 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:20,336 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:20,336 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:20,342 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:20,343 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x5e6a6f3c to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:20,352 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f4eab6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:20,353 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x0a470a3d to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:20,357 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d997c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:20,358 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(139): Connect 0x4bb72437 to 127.0.0.1:63858 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 18:56:20,361 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7993d99e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 18:56:20,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:20,372 DEBUG [hconnection-0x7e4ef054-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:20,372 DEBUG [hconnection-0x13fbea4d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:20,376 DEBUG [hconnection-0x28066334-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:20,376 DEBUG [hconnection-0x76146ff1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:20,377 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:20,380 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33000, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:20,387 DEBUG [hconnection-0x1afc0c73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:20,387 DEBUG [hconnection-0x2627dfe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:20,388 DEBUG [hconnection-0x7263f396-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:20,387 DEBUG [hconnection-0x3bf03c53-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:20,387 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32996, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:20,388 DEBUG [hconnection-0x1115d20a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:20,388 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33008, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:20,388 DEBUG [hconnection-0x564ec7c4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 18:56:20,391 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33016, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:20,392 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:20,392 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33020, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:20,392 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:20,393 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33034, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:20,398 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33038, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 18:56:20,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:20,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:20,400 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:20,400 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:20,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:20,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:20,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,401 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,402 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:20,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:20,402 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:20,403 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:20,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:20,403 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,404 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:20,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:20,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,404 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:20,404 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:20,404 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:20,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:20,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:20,405 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:20,405 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:20,405 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:20,408 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,408 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:20,409 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:20,409 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,409 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:20,409 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:20,409 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,409 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:20,409 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:20,411 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,411 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,411 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,411 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:20,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5144a573[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:20,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:20,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:20,416 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,416 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:20,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:20,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:20,417 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,417 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289040417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289040417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288980409, max timestamp=1680288980419Num uniques 859; 2023-03-31 18:56:20,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289040419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:56:20,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288980409, max timestamp=1680288980419Num uniques 859; 2023-03-31 18:56:20,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:56:20,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289040418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680288980409, max timestamp=1680288980419Num uniques 859; 2023-03-31 18:56:20,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:20,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 18:56:20,421 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:20,421 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:20,421 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:20,421 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:20,421 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:20,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:20,423 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,423 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:20,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,423 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,425 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,428 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:20,428 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:20,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:20,428 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,428 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,428 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:20,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:20,428 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:20,428 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,428 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,428 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,428 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,428 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:20,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:20,428 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:20,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:20,451 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331b5060d06cf784d3ca12dafc96fd3d885 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b5060d06cf784d3ca12dafc96fd3d885 2023-03-31 18:56:20,461 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/44444be64c764613a058cab332105e1f 2023-03-31 18:56:20,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289040463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289040521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289040521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289040522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289040522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,528 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:20,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:20,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:20,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:20,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:20,532 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:20,532 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:20,532 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:20,532 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:20,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,540 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,542 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:20,542 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:20,543 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,543 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:20,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:20,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:20,545 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:20,545 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:20,545 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:20,545 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:20,545 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:20,545 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,545 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:20,546 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:20,546 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:20,547 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,547 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371c4e2e[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:20,549 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:20,549 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,551 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:20,551 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,551 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:20,552 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:20,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:20,555 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:20,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:20,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:20,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:20,555 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:20,557 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,557 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,557 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:20,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:20,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,559 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,562 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:20,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:20,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:20,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:20,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:20,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:20,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,562 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:20,562 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:20,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,562 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:20,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:20,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289040566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,662 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:20,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:20,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:20,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:20,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:20,676 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:20,676 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:20,676 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:20,676 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:20,676 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,677 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,678 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,678 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:20,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:20,678 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,678 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:20,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:20,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:20,687 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,687 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:20,687 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:20,687 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:20,690 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60c1089[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:20,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:20,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,695 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:20,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:20,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:20,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:20,700 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:20,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:20,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:20,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:20,700 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:20,702 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,702 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,702 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,703 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,704 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,709 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,709 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,709 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:20,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:20,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:20,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:20,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:20,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:20,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:20,710 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:20,710 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:20,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289040725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289040725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289040726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289040726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:20,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289040770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,810 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:20,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:20,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:20,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:20,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:20,813 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:20,813 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:20,813 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:20,813 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:20,814 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,814 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,815 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:20,815 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:20,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:20,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:20,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:20,817 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:20,817 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:20,817 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:20,817 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:20,817 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:20,818 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,818 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:20,818 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:20,818 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,820 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,820 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:20,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c20b431[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:20,821 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:20,821 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,822 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:20,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:20,823 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:20,823 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,823 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:20,826 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,826 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:20,826 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,826 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:20,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:20,829 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:20,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:20,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:20,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:20,830 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:20,831 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,831 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,831 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:20,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,832 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:20,833 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:20,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:20,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:20,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:20,836 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:20,836 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:20,837 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:20,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:20,837 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:20,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:20,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:20,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/440737de07014187a68b85a991981ee4 2023-03-31 18:56:20,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/c070c7bdc2314a8f81fdaa04c98dbaa2 2023-03-31 18:56:20,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/44444be64c764613a058cab332105e1f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/44444be64c764613a058cab332105e1f 2023-03-31 18:56:20,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/44444be64c764613a058cab332105e1f, entries=150, sequenceid=15, filesize=25.1 K 2023-03-31 18:56:20,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/440737de07014187a68b85a991981ee4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/440737de07014187a68b85a991981ee4 2023-03-31 18:56:20,903 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/440737de07014187a68b85a991981ee4, entries=150, sequenceid=15, filesize=11.6 K 2023-03-31 18:56:20,904 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/c070c7bdc2314a8f81fdaa04c98dbaa2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c070c7bdc2314a8f81fdaa04c98dbaa2 2023-03-31 18:56:20,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c070c7bdc2314a8f81fdaa04c98dbaa2, entries=150, sequenceid=15, filesize=11.6 K 2023-03-31 18:56:20,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f34b89af9bae1a93ef5b2acb275cb610 in 505ms, sequenceid=15, compaction requested=false 2023-03-31 18:56:20,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:20,937 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:20,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:20,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:20,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:20,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:20,940 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:20,940 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:20,940 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:20,940 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:20,940 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,940 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,941 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,941 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:20,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:20,941 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:20,942 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:20,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:20,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:20,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:20,943 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:20,943 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:20,943 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-31 18:56:20,944 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:20,944 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:20,944 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:20,944 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:20,944 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:20,944 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:20,956 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331206b28b546f744a39aa240301dc6ae35 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331206b28b546f744a39aa240301dc6ae35 2023-03-31 18:56:21,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:21,031 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:21,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289041035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289041036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289041036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289041037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289041074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289041140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289041140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289041140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289041140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289041354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289041354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289041355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289041355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,362 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/df4d8569e3224c71b0924fb4ede7bb6a 2023-03-31 18:56:21,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289041577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289041656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289041657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289041658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:21,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289041659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,772 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/4f5f893647a3495f98da3f48697bee36 2023-03-31 18:56:21,782 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/2ee9677781ef4475b64d812daa1e3db0 2023-03-31 18:56:21,786 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/df4d8569e3224c71b0924fb4ede7bb6a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df4d8569e3224c71b0924fb4ede7bb6a 2023-03-31 18:56:21,790 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df4d8569e3224c71b0924fb4ede7bb6a, entries=150, sequenceid=49, filesize=25.1 K 2023-03-31 18:56:21,791 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/4f5f893647a3495f98da3f48697bee36 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/4f5f893647a3495f98da3f48697bee36 2023-03-31 18:56:21,795 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/4f5f893647a3495f98da3f48697bee36, entries=150, sequenceid=49, filesize=11.6 K 2023-03-31 18:56:21,795 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/2ee9677781ef4475b64d812daa1e3db0 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2ee9677781ef4475b64d812daa1e3db0 2023-03-31 18:56:21,798 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2ee9677781ef4475b64d812daa1e3db0, entries=150, sequenceid=49, filesize=11.6 K 2023-03-31 18:56:21,799 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for f34b89af9bae1a93ef5b2acb275cb610 in 856ms, sequenceid=49, compaction requested=false 2023-03-31 18:56:21,799 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:21,799 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:21,802 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@383beffb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:21,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:21,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:21,806 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:21,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:21,807 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,807 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:21,808 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,808 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:21,808 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:21,808 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:21,811 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:21,811 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:21,811 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:21,812 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:21,812 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:21,813 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:21,813 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:21,813 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:21,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:21,814 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,815 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:21,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:21,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:21,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:21,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:21,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:21,818 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:21,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:21,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:21,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:21,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:21,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:21,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:21,819 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:21,819 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:21,919 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:21,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:21,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:21,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:21,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:21,923 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:21,923 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:21,923 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:21,923 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:21,923 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:21,923 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:21,925 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:21,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:21,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:21,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:21,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:21,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:21,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:21,927 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:21,927 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:21,927 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:56:21,927 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:21,927 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:21,927 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:21,927 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:21,927 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:21,927 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:21,947 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033166a52b03fc6147c09d4409161884d113 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033166a52b03fc6147c09d4409161884d113 2023-03-31 18:56:21,954 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/8a056cb0bbf84b919ec78a70fcd3e823 2023-03-31 18:56:21,970 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/1878856c38e14c2e9a444b470bcb1637 2023-03-31 18:56:22,020 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 18:56:22,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:22,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:22,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289042178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289042179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289042180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288982171, max timestamp=1680288982180Num uniques 859; 2023-03-31 18:56:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288982171, max timestamp=1680288982180Num uniques 859; 2023-03-31 18:56:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680288982171, max timestamp=1680288982180Num uniques 859; 2023-03-31 18:56:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 18:56:22,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289042231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289042284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289042284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289042284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289042334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,386 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/9e3570ff6a154ee9ab13f8b6f2c64992 2023-03-31 18:56:22,390 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/8a056cb0bbf84b919ec78a70fcd3e823 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8a056cb0bbf84b919ec78a70fcd3e823 2023-03-31 18:56:22,393 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8a056cb0bbf84b919ec78a70fcd3e823, entries=100, sequenceid=60, filesize=18.4 K 2023-03-31 18:56:22,394 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/1878856c38e14c2e9a444b470bcb1637 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1878856c38e14c2e9a444b470bcb1637 2023-03-31 18:56:22,397 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1878856c38e14c2e9a444b470bcb1637, entries=100, sequenceid=60, filesize=9.3 K 2023-03-31 18:56:22,397 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/9e3570ff6a154ee9ab13f8b6f2c64992 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9e3570ff6a154ee9ab13f8b6f2c64992 2023-03-31 18:56:22,401 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9e3570ff6a154ee9ab13f8b6f2c64992, entries=100, sequenceid=60, filesize=9.3 K 2023-03-31 18:56:22,401 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f34b89af9bae1a93ef5b2acb275cb610 in 474ms, sequenceid=60, compaction requested=true 2023-03-31 18:56:22,401 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:22,401 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:22,403 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d03bbec[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:22,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:22,405 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,406 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,406 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:22,406 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:22,406 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:22,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,406 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:22,409 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:22,409 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,409 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,409 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:22,413 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:22,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:22,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:22,413 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:22,413 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:22,414 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,414 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,415 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:22,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:22,415 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,416 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,420 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:22,420 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:22,421 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:22,420 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:22,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:22,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:22,420 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,421 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,421 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,421 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,421 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,421 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:22,421 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:22,421 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:22,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:22,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:22,491 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-31 18:56:22,492 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:22,492 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:22,492 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:22,492 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:22,492 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:22,492 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:22,503 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331695a46b3408248bcbc2cb0c553c7e529 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331695a46b3408248bcbc2cb0c553c7e529 2023-03-31 18:56:22,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289042502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289042502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289042502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,521 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:22,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:22,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:22,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:22,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:22,524 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:22,524 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:22,524 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:22,524 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:22,524 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,524 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,527 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:22,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:22,527 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:22,528 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,528 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:22,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:22,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:22,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:22,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:22,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:22,529 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:22,531 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,531 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,531 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,531 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bed669d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:22,533 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:22,533 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,534 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:22,534 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,534 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:22,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:22,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:22,536 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,536 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:22,538 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:22,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:22,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:22,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:22,538 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:22,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289042538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,541 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,541 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,541 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:22,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:22,541 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,543 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,545 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,545 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,545 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:22,545 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:22,546 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:22,546 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,546 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,546 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,546 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,545 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:22,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:22,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:22,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:22,546 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:22,546 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:22,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:22,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289042587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289042607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289042607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289042607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,646 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:22,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:22,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:22,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:22,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:22,649 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:22,650 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:22,650 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:22,650 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:22,650 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,650 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,651 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:22,651 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:22,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,652 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:22,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:22,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:22,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:22,653 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:22,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:22,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:22,654 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:22,654 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:22,654 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:22,654 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:22,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:22,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:22,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:22,654 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:22,655 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4225daed[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:22,657 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:22,658 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,660 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,660 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:22,660 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:22,660 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:22,660 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,660 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:22,661 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:22,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,662 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:22,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:22,664 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:22,665 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:22,665 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:22,665 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:22,665 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:22,666 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,666 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:22,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,666 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:22,667 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,668 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:22,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:22,670 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:22,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,670 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:22,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:22,671 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:22,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:22,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:22,671 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,671 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:22,671 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:22,771 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:22,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:22,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:22,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:22,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:22,774 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:22,774 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:22,774 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:22,774 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:22,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,775 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,777 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:22,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:22,777 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,778 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:22,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:22,779 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:22,779 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:22,779 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:22,779 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:22,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,782 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,782 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:22,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2586a165[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:22,783 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:22,783 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,784 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,784 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:22,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,785 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:22,785 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:22,785 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:22,786 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:22,789 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:22,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:22,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:22,789 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:22,789 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:22,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:22,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,791 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:22,792 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,793 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,795 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:22,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:22,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:22,796 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:22,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:22,796 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:22,796 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:22,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:22,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:22,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:22,796 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289042809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289042811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289042811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:22,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289042843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,896 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:22,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:22,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:22,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:22,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:22,899 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:22,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:22,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:22,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:22,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,901 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:22,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:22,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:22,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:22,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:22,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:22,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:22,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:22,903 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:22,903 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:22,903 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:22,903 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:22,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:22,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:22,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:22,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:22,909 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/c1ff0a7c8a90422d9cd8a38a5b064a84 2023-03-31 18:56:22,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,912 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:22,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@552ab2e2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:22,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:22,915 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,916 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:22,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:22,918 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:22,922 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:22,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:22,922 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:22,923 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:22,923 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:22,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:22,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:22,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:22,925 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:22,926 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:22,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:22,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:22,928 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:22,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:22,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:22,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:22,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:22,928 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:22,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:22,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:22,929 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:22,929 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:22,929 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,029 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:23,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:23,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:23,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:23,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:23,033 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:23,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:23,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:23,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,036 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:23,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:23,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:23,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:23,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:23,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:23,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:23,038 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,039 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:23,039 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:23,039 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:23,040 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:23,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3593ff54[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:23,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:23,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,043 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:23,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:23,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:23,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,045 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:23,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:23,048 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:23,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:23,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:23,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:23,048 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:23,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,052 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,053 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:23,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:23,056 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:23,056 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:23,056 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:23,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:23,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289043113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:23,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:23,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289043115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289043115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,157 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:23,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:23,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:23,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:23,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:23,160 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:23,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:23,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:23,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,162 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:23,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:23,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:23,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:23,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:23,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:23,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:23,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:23,164 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:23,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:23,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:23,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:23,166 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@590da9b9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:23,167 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:23,167 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,169 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:23,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,170 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:23,171 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:23,171 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,171 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,171 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:23,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:23,174 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:23,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:23,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:23,174 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:23,174 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:23,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,176 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,178 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,180 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:23,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:23,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,180 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:23,180 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:23,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,180 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,181 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,280 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:23,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:23,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:23,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:23,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:23,284 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:23,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:23,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:23,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,284 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,287 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:23,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:23,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:23,288 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:23,289 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,289 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:23,289 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:23,289 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:23,290 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,291 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,291 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:23,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4910b674[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:23,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:23,292 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,296 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:23,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:23,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:23,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,296 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:23,297 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,297 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:23,297 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,297 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:23,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:23,300 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:23,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:23,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:23,300 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:23,300 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:23,301 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,302 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,303 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,306 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:23,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:23,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,306 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,307 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:23,307 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:23,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:23,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/e36fd4df41c6493aa474b2ee1ff44509 2023-03-31 18:56:23,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:23,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289043348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,407 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:23,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:23,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:23,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:23,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:23,410 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:23,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:23,410 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:23,411 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,411 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,412 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,412 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,412 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,412 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:23,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:23,414 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,414 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:23,414 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:23,414 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:23,416 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492da016[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:23,417 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:23,417 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,419 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,419 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:23,419 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:23,419 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,419 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:23,419 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:23,422 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:23,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:23,425 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:23,425 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:23,425 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:23,425 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:23,425 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:23,427 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,427 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,427 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,428 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,429 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,431 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,431 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,431 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,431 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,431 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,432 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:23,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:23,431 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:23,432 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,432 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,432 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,432 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:23,432 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:23,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,532 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:23,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:23,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:23,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:23,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:23,536 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:23,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:23,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:23,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,536 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,538 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,538 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,539 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,539 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:23,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:23,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:23,540 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,540 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:23,540 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:23,540 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:23,540 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:23,543 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,543 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,544 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,544 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@636e8f38[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:23,545 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:23,545 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,547 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:23,547 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,547 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:23,548 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:23,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,548 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:23,551 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:23,551 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:23,551 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:23,551 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:23,551 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:23,553 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,553 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,554 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,554 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,555 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,558 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,558 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,558 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,558 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,558 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:23,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:23,558 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,558 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,559 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,559 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:23,559 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:23,559 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,559 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:23,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289043615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:23,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289043620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:23,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289043623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,659 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:23,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:23,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:23,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:23,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:23,677 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:23,678 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,678 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:23,678 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:23,678 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,678 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,680 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,680 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,680 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,680 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:23,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:23,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:23,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:23,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:23,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,681 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:23,682 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,682 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:23,682 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:23,682 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:23,683 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:23,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@579ab28d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:23,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:23,685 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,686 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:23,687 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,687 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,692 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:23,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:23,694 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:23,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:23,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:23,694 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:23,695 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:23,696 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,696 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,696 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,697 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,698 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,700 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,700 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:23,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:23,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:23,701 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:23,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,701 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:23,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/60953f515e8b451682dc8c67725279ed 2023-03-31 18:56:23,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/c1ff0a7c8a90422d9cd8a38a5b064a84 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c1ff0a7c8a90422d9cd8a38a5b064a84 2023-03-31 18:56:23,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c1ff0a7c8a90422d9cd8a38a5b064a84, entries=200, sequenceid=97, filesize=31.9 K 2023-03-31 18:56:23,751 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/e36fd4df41c6493aa474b2ee1ff44509 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e36fd4df41c6493aa474b2ee1ff44509 2023-03-31 18:56:23,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e36fd4df41c6493aa474b2ee1ff44509, entries=150, sequenceid=97, filesize=11.6 K 2023-03-31 18:56:23,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/60953f515e8b451682dc8c67725279ed as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/60953f515e8b451682dc8c67725279ed 2023-03-31 18:56:23,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/60953f515e8b451682dc8c67725279ed, entries=150, sequenceid=97, filesize=11.6 K 2023-03-31 18:56:23,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for f34b89af9bae1a93ef5b2acb275cb610 in 1268ms, sequenceid=97, compaction requested=true 2023-03-31 18:56:23,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:23,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:23,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:23,758 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:23,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:23,758 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:23,759 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 102976 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:23,760 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:23,760 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/A is initiating minor compaction (all files) 2023-03-31 18:56:23,760 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/B is initiating minor compaction (all files) 2023-03-31 18:56:23,760 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/B in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,760 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/A in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,760 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/440737de07014187a68b85a991981ee4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/4f5f893647a3495f98da3f48697bee36, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1878856c38e14c2e9a444b470bcb1637, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e36fd4df41c6493aa474b2ee1ff44509] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=44.1 K 2023-03-31 18:56:23,760 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/44444be64c764613a058cab332105e1f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df4d8569e3224c71b0924fb4ede7bb6a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8a056cb0bbf84b919ec78a70fcd3e823, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c1ff0a7c8a90422d9cd8a38a5b064a84] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=100.6 K 2023-03-31 18:56:23,760 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 440737de07014187a68b85a991981ee4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680288980401 2023-03-31 18:56:23,761 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 44444be64c764613a058cab332105e1f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680288980401 2023-03-31 18:56:23,761 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4f5f893647a3495f98da3f48697bee36, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680288980446 2023-03-31 18:56:23,761 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting df4d8569e3224c71b0924fb4ede7bb6a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680288980446 2023-03-31 18:56:23,761 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8a056cb0bbf84b919ec78a70fcd3e823, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680288981035 2023-03-31 18:56:23,761 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 1878856c38e14c2e9a444b470bcb1637, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680288981035 2023-03-31 18:56:23,762 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c1ff0a7c8a90422d9cd8a38a5b064a84, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680288982214 2023-03-31 18:56:23,762 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e36fd4df41c6493aa474b2ee1ff44509, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680288982229 2023-03-31 18:56:23,771 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#B#compaction#540 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:23,774 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#A#compaction#539 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:23,801 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:23,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:23,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:23,805 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:23,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:23,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:23,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,805 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,808 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,808 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,808 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,808 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:23,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:23,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:23,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:23,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:23,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:23,810 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:23,810 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:23,810 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,810 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:23,810 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:56:23,810 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:23,810 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:23,810 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:23,810 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:23,810 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:23,810 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:23,837 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331119e0eb14a18442aa543599fc5e87f7e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331119e0eb14a18442aa543599fc5e87f7e 2023-03-31 18:56:23,846 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/69cbc3672fc14d56a7c5ff1ba74cb104 2023-03-31 18:56:23,857 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/ee4093a270c648cba1fff0cf834d6511 2023-03-31 18:56:23,878 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/632780cef648409eaa7b9ab0fe4e5300 2023-03-31 18:56:23,884 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/69cbc3672fc14d56a7c5ff1ba74cb104 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/69cbc3672fc14d56a7c5ff1ba74cb104 2023-03-31 18:56:23,887 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/69cbc3672fc14d56a7c5ff1ba74cb104, entries=150, sequenceid=107, filesize=25.1 K 2023-03-31 18:56:23,888 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/ee4093a270c648cba1fff0cf834d6511 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ee4093a270c648cba1fff0cf834d6511 2023-03-31 18:56:23,891 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ee4093a270c648cba1fff0cf834d6511, entries=150, sequenceid=107, filesize=11.6 K 2023-03-31 18:56:23,892 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/632780cef648409eaa7b9ab0fe4e5300 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/632780cef648409eaa7b9ab0fe4e5300 2023-03-31 18:56:23,895 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/632780cef648409eaa7b9ab0fe4e5300, entries=150, sequenceid=107, filesize=11.6 K 2023-03-31 18:56:23,896 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for f34b89af9bae1a93ef5b2acb275cb610 in 86ms, sequenceid=107, compaction requested=true 2023-03-31 18:56:23,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:23,896 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:23,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:23,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:23,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:23,896 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:23,898 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf50d1c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:23,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:23,900 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,901 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:23,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:23,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:23,901 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:23,903 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:23,906 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:23,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:23,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:23,906 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:23,906 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:23,908 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,908 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,908 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:23,909 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:23,910 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:23,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:23,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:23,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:23,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:23,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:23,913 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:23,913 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:23,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:23,913 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:23,912 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:23,913 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:23,913 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,913 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:23,913 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,013 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:24,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:24,020 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:24,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:24,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:24,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,022 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,022 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:24,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:24,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:24,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:24,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:24,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:24,034 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,034 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:24,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7c2364[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:24,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:24,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,038 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:24,039 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,039 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,040 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,040 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:24,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:24,043 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:24,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:24,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:24,043 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:24,043 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:24,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,045 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:24,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,050 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,051 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:24,050 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,051 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:24,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,151 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:24,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:24,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:24,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:24,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:24,156 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:24,156 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,156 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:24,156 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:24,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,157 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,159 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,159 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:24,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:24,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:24,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:24,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:24,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:24,161 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,162 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:24,162 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:24,162 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:24,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,163 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28fc7cfc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:24,166 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:24,166 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,167 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:24,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:24,167 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:24,167 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,167 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:24,169 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:24,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:24,172 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:24,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:24,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:24,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:24,172 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:24,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,175 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,176 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:24,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:24,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,183 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:24,183 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:24,182 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,186 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/f4f96058fd6e445a858c63a2da5abc24 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f4f96058fd6e445a858c63a2da5abc24 2023-03-31 18:56:24,191 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/B of f34b89af9bae1a93ef5b2acb275cb610 into f4f96058fd6e445a858c63a2da5abc24(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:24,192 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:24,192 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/B, priority=12, startTime=1680288983758; duration=0sec 2023-03-31 18:56:24,192 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:24,192 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:56:24,193 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:56:24,193 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/C is initiating minor compaction (all files) 2023-03-31 18:56:24,193 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/C in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,194 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c070c7bdc2314a8f81fdaa04c98dbaa2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2ee9677781ef4475b64d812daa1e3db0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9e3570ff6a154ee9ab13f8b6f2c64992, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/60953f515e8b451682dc8c67725279ed, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/632780cef648409eaa7b9ab0fe4e5300] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=55.7 K 2023-03-31 18:56:24,194 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting c070c7bdc2314a8f81fdaa04c98dbaa2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680288980401 2023-03-31 18:56:24,194 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2ee9677781ef4475b64d812daa1e3db0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680288980446 2023-03-31 18:56:24,195 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9e3570ff6a154ee9ab13f8b6f2c64992, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680288981035 2023-03-31 18:56:24,195 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 60953f515e8b451682dc8c67725279ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680288982229 2023-03-31 18:56:24,195 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 632780cef648409eaa7b9ab0fe4e5300, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680288982496 2023-03-31 18:56:24,203 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#C#compaction#544 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:24,211 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/cb156128cd3c4bea808718aa8c63efb6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/cb156128cd3c4bea808718aa8c63efb6 2023-03-31 18:56:24,215 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/C of f34b89af9bae1a93ef5b2acb275cb610 into cb156128cd3c4bea808718aa8c63efb6(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:24,215 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:24,215 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/C, priority=11, startTime=1680288983758; duration=0sec 2023-03-31 18:56:24,215 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:24,283 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:24,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:24,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:24,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:24,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:24,287 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:24,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:24,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:24,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,291 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,291 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,291 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,291 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:24,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:24,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:24,293 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:24,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:24,294 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:24,295 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:24,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a70bdb4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:24,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:24,298 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,299 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:24,299 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,299 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:24,302 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:24,304 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:24,304 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:24,305 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:24,305 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:24,305 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:24,307 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,308 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:24,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:24,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,320 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:24,320 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:24,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,320 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,321 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,321 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:24,390 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:56:24,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:24,390 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:24,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:24,390 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:24,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:24,390 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:24,405 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303315197a427b1cf46408d74f040ec297c85 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315197a427b1cf46408d74f040ec297c85 2023-03-31 18:56:24,411 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=122, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/63bab03e5eaf49dd970805ee7c6570a3 2023-03-31 18:56:24,420 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:24,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:24,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:24,424 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:24,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:24,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:24,424 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,425 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,426 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,426 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,426 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,426 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:24,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:24,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:24,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:24,427 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,428 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:24,428 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:24,428 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:24,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,431 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,431 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19dec80b[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:24,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:24,433 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,434 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:24,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:24,434 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:24,434 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,434 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:24,436 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:24,436 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,436 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,436 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:24,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:24,439 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:24,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:24,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:24,439 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:24,439 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:24,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,441 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,443 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:24,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:24,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,446 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:24,446 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:24,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,447 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288984428, max timestamp=1680288984451Num uniques 859; 2023-03-31 18:56:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288984428, max timestamp=1680288984451Num uniques 859; 2023-03-31 18:56:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:24,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288984428, max timestamp=1680288984451Num uniques 859; 2023-03-31 18:56:24,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:24,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:24,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289044505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,547 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:24,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:24,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:24,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:24,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:24,550 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:24,550 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,550 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:24,550 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:24,550 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,550 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,551 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,551 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,552 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,552 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:24,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:24,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:24,553 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,553 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:24,553 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:24,554 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:24,555 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,555 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,555 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,555 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4767af57[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:24,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:24,557 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,558 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:24,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:24,558 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:24,558 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,558 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:24,560 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:24,561 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,561 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,561 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:24,563 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:24,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:24,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:24,563 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:24,563 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:24,564 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,564 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,564 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,566 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:24,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:24,569 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:24,569 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:24,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:24,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,569 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,597 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/4f662ded670b45859488f567fa1892dd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4f662ded670b45859488f567fa1892dd 2023-03-31 18:56:24,601 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/A of f34b89af9bae1a93ef5b2acb275cb610 into 4f662ded670b45859488f567fa1892dd(size=25.3 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:24,601 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:24,601 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/A, priority=12, startTime=1680288983758; duration=0sec 2023-03-31 18:56:24,601 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:24,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:24,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289044601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,604 DEBUG [Thread-3975] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:56:24,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:24,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289044609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:24,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289044620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:24,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289044625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:24,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289044634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,669 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:24,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:24,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:24,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:24,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:24,673 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:24,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:24,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:24,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,673 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,676 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,676 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,676 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,676 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:24,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:24,677 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:24,677 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,677 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:24,678 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,678 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:24,678 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:24,678 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:24,680 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:24,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c3730db[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:24,682 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:24,682 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,683 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:24,684 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,684 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:24,685 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,685 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:24,685 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,685 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:24,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:24,688 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:24,688 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:24,688 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:24,688 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:24,688 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:24,691 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,691 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,691 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,692 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,693 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,695 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,695 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,696 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,696 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,696 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:24,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:24,696 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,696 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,696 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,696 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:24,696 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,696 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,696 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:24,797 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:24,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:24,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:24,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:24,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:24,800 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:24,800 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,800 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:24,800 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:24,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,802 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,803 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,803 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,803 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,803 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:24,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:24,805 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,805 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:24,805 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:24,805 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:24,807 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:24,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2305a1f4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:24,809 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:24,809 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:24,812 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,812 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:24,813 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:24,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289044813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:24,816 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:24,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:24,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:24,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:24,816 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:24,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:24,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:24,820 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:24,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/6fb70033454e4d77ad253b7b6d79d9cb 2023-03-31 18:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,821 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:24,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:24,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:24,824 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:24,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:24,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,825 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:24,825 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:24,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:24,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/3110c06252a14e8aa9b68f26bd5060b7 2023-03-31 18:56:24,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/63bab03e5eaf49dd970805ee7c6570a3 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/63bab03e5eaf49dd970805ee7c6570a3 2023-03-31 18:56:24,839 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/63bab03e5eaf49dd970805ee7c6570a3, entries=200, sequenceid=122, filesize=31.9 K 2023-03-31 18:56:24,840 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/6fb70033454e4d77ad253b7b6d79d9cb as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/6fb70033454e4d77ad253b7b6d79d9cb 2023-03-31 18:56:24,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/6fb70033454e4d77ad253b7b6d79d9cb, entries=150, sequenceid=122, filesize=11.6 K 2023-03-31 18:56:24,844 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/3110c06252a14e8aa9b68f26bd5060b7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3110c06252a14e8aa9b68f26bd5060b7 2023-03-31 18:56:24,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3110c06252a14e8aa9b68f26bd5060b7, entries=150, sequenceid=122, filesize=11.6 K 2023-03-31 18:56:24,849 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f34b89af9bae1a93ef5b2acb275cb610 in 459ms, sequenceid=122, compaction requested=true 2023-03-31 18:56:24,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:24,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:24,849 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:24,849 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:24,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:24,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:24,850 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84316 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:24,850 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:24,851 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/A is initiating minor compaction (all files) 2023-03-31 18:56:24,851 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/B is initiating minor compaction (all files) 2023-03-31 18:56:24,851 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/A in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,851 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/B in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,851 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4f662ded670b45859488f567fa1892dd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/69cbc3672fc14d56a7c5ff1ba74cb104, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/63bab03e5eaf49dd970805ee7c6570a3] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=82.3 K 2023-03-31 18:56:24,851 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f4f96058fd6e445a858c63a2da5abc24, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ee4093a270c648cba1fff0cf834d6511, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/6fb70033454e4d77ad253b7b6d79d9cb] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=35.0 K 2023-03-31 18:56:24,851 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 4f662ded670b45859488f567fa1892dd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680288982229 2023-03-31 18:56:24,851 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f4f96058fd6e445a858c63a2da5abc24, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680288982229 2023-03-31 18:56:24,851 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 69cbc3672fc14d56a7c5ff1ba74cb104, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680288982496 2023-03-31 18:56:24,852 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ee4093a270c648cba1fff0cf834d6511, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680288982496 2023-03-31 18:56:24,852 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 63bab03e5eaf49dd970805ee7c6570a3, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680288984365 2023-03-31 18:56:24,852 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 6fb70033454e4d77ad253b7b6d79d9cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680288984371 2023-03-31 18:56:24,862 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#B#compaction#549 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:24,865 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#A#compaction#548 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:24,925 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:24,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:24,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:24,928 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/5603991593534077ab3db86b006193bb as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5603991593534077ab3db86b006193bb 2023-03-31 18:56:24,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:24,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:24,930 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:24,930 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,930 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:24,930 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:24,931 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,931 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,933 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:24,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:24,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:24,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:24,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:24,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:24,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:24,936 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/A of f34b89af9bae1a93ef5b2acb275cb610 into 5603991593534077ab3db86b006193bb(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:24,936 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:24,936 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/A, priority=13, startTime=1680288984849; duration=0sec 2023-03-31 18:56:24,936 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:24,936 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 18:56:24,937 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 18:56:24,937 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 18:56:24,937 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. because compaction request was cancelled 2023-03-31 18:56:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:24,937 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:24,937 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:24,938 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-31 18:56:24,938 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:24,938 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:24,938 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:24,938 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:24,938 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:24,938 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:24,965 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331fd0c665423444b71aa7f7bb0a253ba8b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331fd0c665423444b71aa7f7bb0a253ba8b 2023-03-31 18:56:24,971 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=161, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/8f6c630ec1de463799da3289f694ba90 2023-03-31 18:56:25,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:25,118 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:25,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:25,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289045137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289045240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,289 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/0b4d88e895b843d098f475c93632c491 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/0b4d88e895b843d098f475c93632c491 2023-03-31 18:56:25,293 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/B of f34b89af9bae1a93ef5b2acb275cb610 into 0b4d88e895b843d098f475c93632c491(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:25,293 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:25,293 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/B, priority=13, startTime=1680288984849; duration=0sec 2023-03-31 18:56:25,293 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:25,381 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/73f6b473ebd34e89b66b065a69ced439 2023-03-31 18:56:25,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:25,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289045444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:25,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33034 deadline: 1680289045746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,791 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/3ec5ca73b3284d9eb23e48e03b2693bd 2023-03-31 18:56:25,795 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/8f6c630ec1de463799da3289f694ba90 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8f6c630ec1de463799da3289f694ba90 2023-03-31 18:56:25,798 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8f6c630ec1de463799da3289f694ba90, entries=150, sequenceid=161, filesize=25.3 K 2023-03-31 18:56:25,799 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/73f6b473ebd34e89b66b065a69ced439 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73f6b473ebd34e89b66b065a69ced439 2023-03-31 18:56:25,802 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73f6b473ebd34e89b66b065a69ced439, entries=150, sequenceid=161, filesize=11.8 K 2023-03-31 18:56:25,803 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/3ec5ca73b3284d9eb23e48e03b2693bd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3ec5ca73b3284d9eb23e48e03b2693bd 2023-03-31 18:56:25,806 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3ec5ca73b3284d9eb23e48e03b2693bd, entries=150, sequenceid=161, filesize=11.8 K 2023-03-31 18:56:25,806 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for f34b89af9bae1a93ef5b2acb275cb610 in 869ms, sequenceid=161, compaction requested=true 2023-03-31 18:56:25,807 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:25,807 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:25,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:25,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:25,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:25,807 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:25,809 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:25,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:25,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c74ab0f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:25,811 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:25,811 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:25,812 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:25,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:25,812 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,813 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:25,812 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:25,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:25,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:25,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,814 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:25,814 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:25,814 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:25,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:25,816 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:25,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:25,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:25,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:25,816 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:25,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:25,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:25,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:25,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:25,819 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:25,821 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,823 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:25,823 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:25,823 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:25,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:25,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:25,823 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:25,823 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:25,823 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:25,823 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:25,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:25,823 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:25,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:25,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:25,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:25,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:25,924 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:25,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:25,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:25,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:25,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:25,927 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:25,927 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:25,928 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:25,928 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:25,928 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:25,928 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:25,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,933 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:25,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:25,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:25,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:25,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:25,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:25,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:25,935 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:25,935 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:25,935 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:56:25,935 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:25,935 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:25,936 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:25,936 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:25,936 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:25,936 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:26,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:26,252 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:26,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288986299, max timestamp=1680288986307Num uniques 859; 2023-03-31 18:56:26,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:26,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288986299, max timestamp=1680288986307Num uniques 859; 2023-03-31 18:56:26,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:26,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288986299, max timestamp=1680288986307Num uniques 859; 2023-03-31 18:56:26,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:26,350 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033147e57dcb32b444f79afa465a92320c99 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033147e57dcb32b444f79afa465a92320c99 2023-03-31 18:56:26,363 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=172, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/d25b046269544aaab3604ef08d93cdb8 2023-03-31 18:56:26,373 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/eb56197c414f499b93c496092c7067c8 2023-03-31 18:56:26,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:26,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289046385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:26,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289046489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:26,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289046625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,629 DEBUG [Thread-3971] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4125 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:56:26,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:26,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289046642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,646 DEBUG [Thread-3977] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:56:26,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:26,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289046648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,650 DEBUG [Thread-3973] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:56:26,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:26,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289046693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,788 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/c6ff96acc3ab4838b57e2dba6abb879b 2023-03-31 18:56:26,792 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/d25b046269544aaab3604ef08d93cdb8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/d25b046269544aaab3604ef08d93cdb8 2023-03-31 18:56:26,795 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/d25b046269544aaab3604ef08d93cdb8, entries=150, sequenceid=172, filesize=25.3 K 2023-03-31 18:56:26,804 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/eb56197c414f499b93c496092c7067c8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/eb56197c414f499b93c496092c7067c8 2023-03-31 18:56:26,808 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/eb56197c414f499b93c496092c7067c8, entries=150, sequenceid=172, filesize=11.8 K 2023-03-31 18:56:26,809 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/c6ff96acc3ab4838b57e2dba6abb879b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c6ff96acc3ab4838b57e2dba6abb879b 2023-03-31 18:56:26,812 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c6ff96acc3ab4838b57e2dba6abb879b, entries=150, sequenceid=172, filesize=11.8 K 2023-03-31 18:56:26,813 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f34b89af9bae1a93ef5b2acb275cb610 in 878ms, sequenceid=172, compaction requested=true 2023-03-31 18:56:26,813 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:26,813 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:26,814 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:26,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:26,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3693b8c1[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:26,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:26,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:26,817 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:26,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:26,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:26,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:26,818 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:26,818 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,818 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:26,820 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:26,823 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:26,823 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:26,823 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:26,823 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:26,823 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:26,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:26,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:26,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:26,824 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:26,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:26,825 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:26,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:26,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:26,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:26,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:26,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:26,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:26,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:26,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:26,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:26,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:26,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:26,829 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:26,829 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:26,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:26,929 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:26,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:26,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:26,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:26,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:26,933 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:26,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:26,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:26,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:26,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:26,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:26,935 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:26,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:26,935 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:26,935 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:26,935 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:26,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:26,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:26,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:26,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:26,937 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-31 18:56:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:26,938 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:26,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:26,999 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:27,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:27,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289047021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:27,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:27,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289047126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:27,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:27,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289047330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:27,352 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331b0e804afc7f44eb2894eaf4b8b501a20 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b0e804afc7f44eb2894eaf4b8b501a20 2023-03-31 18:56:27,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:27,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289047633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:27,758 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=209, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/5a43151c0c0c4b7f86fda16adc83e87f 2023-03-31 18:56:28,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:28,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289048138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,168 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/9ff411089d434bd68a8677d942d32738 2023-03-31 18:56:28,578 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/d7067f38ba654fa690b4ec6200d16112 2023-03-31 18:56:28,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/5a43151c0c0c4b7f86fda16adc83e87f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5a43151c0c0c4b7f86fda16adc83e87f 2023-03-31 18:56:28,587 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5a43151c0c0c4b7f86fda16adc83e87f, entries=150, sequenceid=209, filesize=25.3 K 2023-03-31 18:56:28,587 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/9ff411089d434bd68a8677d942d32738 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9ff411089d434bd68a8677d942d32738 2023-03-31 18:56:28,591 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9ff411089d434bd68a8677d942d32738, entries=150, sequenceid=209, filesize=11.8 K 2023-03-31 18:56:28,591 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/d7067f38ba654fa690b4ec6200d16112 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d7067f38ba654fa690b4ec6200d16112 2023-03-31 18:56:28,595 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d7067f38ba654fa690b4ec6200d16112, entries=150, sequenceid=209, filesize=11.8 K 2023-03-31 18:56:28,595 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for f34b89af9bae1a93ef5b2acb275cb610 in 1658ms, sequenceid=209, compaction requested=true 2023-03-31 18:56:28,595 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:28,595 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:28,598 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317e720[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:28,600 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:28,600 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,601 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:28,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:28,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:28,601 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,601 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:28,602 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:28,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:28,606 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:28,606 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:28,606 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:28,606 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:28,606 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:28,607 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,607 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,607 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:28,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:28,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,608 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,610 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,613 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:28,613 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,613 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:28,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:28,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:28,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:28,613 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,614 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,614 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,614 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,614 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,613 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:28,614 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,614 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:28,614 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:28,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:28,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:28,617 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:56:28,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:28,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:28,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:28,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:28,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:28,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:28,636 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331c4f2a84cc83e4641887ef3712037629f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c4f2a84cc83e4641887ef3712037629f 2023-03-31 18:56:28,646 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=220, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/b115b79ab175439687681515cceacf31 2023-03-31 18:56:28,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/2e34044873724ad79d7fcae000cf643d 2023-03-31 18:56:28,681 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/7cf4e3090cbf4beaba15bdc46b9a9bf1 2023-03-31 18:56:28,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/b115b79ab175439687681515cceacf31 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/b115b79ab175439687681515cceacf31 2023-03-31 18:56:28,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/b115b79ab175439687681515cceacf31, entries=150, sequenceid=220, filesize=25.3 K 2023-03-31 18:56:28,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/2e34044873724ad79d7fcae000cf643d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2e34044873724ad79d7fcae000cf643d 2023-03-31 18:56:28,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2e34044873724ad79d7fcae000cf643d, entries=150, sequenceid=220, filesize=11.8 K 2023-03-31 18:56:28,692 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/7cf4e3090cbf4beaba15bdc46b9a9bf1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/7cf4e3090cbf4beaba15bdc46b9a9bf1 2023-03-31 18:56:28,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/7cf4e3090cbf4beaba15bdc46b9a9bf1, entries=150, sequenceid=220, filesize=11.8 K 2023-03-31 18:56:28,696 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=107.34 KB/109920 for f34b89af9bae1a93ef5b2acb275cb610 in 79ms, sequenceid=220, compaction requested=true 2023-03-31 18:56:28,696 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:28,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:28,696 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:56:28,696 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 18:56:28,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:28,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:28,697 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 129570 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:56:28,697 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/A is initiating minor compaction (all files) 2023-03-31 18:56:28,697 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/A in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:28,697 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5603991593534077ab3db86b006193bb, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8f6c630ec1de463799da3289f694ba90, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/d25b046269544aaab3604ef08d93cdb8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5a43151c0c0c4b7f86fda16adc83e87f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/b115b79ab175439687681515cceacf31] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=126.5 K 2023-03-31 18:56:28,698 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60270 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 18:56:28,698 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/B is initiating minor compaction (all files) 2023-03-31 18:56:28,698 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/B in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:28,698 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/0b4d88e895b843d098f475c93632c491, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73f6b473ebd34e89b66b065a69ced439, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/eb56197c414f499b93c496092c7067c8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9ff411089d434bd68a8677d942d32738, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2e34044873724ad79d7fcae000cf643d] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=58.9 K 2023-03-31 18:56:28,698 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 5603991593534077ab3db86b006193bb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680288984371 2023-03-31 18:56:28,698 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 0b4d88e895b843d098f475c93632c491, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680288984371 2023-03-31 18:56:28,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:28,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:56:28,699 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 73f6b473ebd34e89b66b065a69ced439, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1680288984497 2023-03-31 18:56:28,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:28,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:28,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:28,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:28,699 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8f6c630ec1de463799da3289f694ba90, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1680288984497 2023-03-31 18:56:28,699 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting eb56197c414f499b93c496092c7067c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680288985123 2023-03-31 18:56:28,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:28,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:28,699 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting d25b046269544aaab3604ef08d93cdb8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680288985123 2023-03-31 18:56:28,699 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9ff411089d434bd68a8677d942d32738, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680288986343 2023-03-31 18:56:28,700 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 5a43151c0c0c4b7f86fda16adc83e87f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680288986343 2023-03-31 18:56:28,700 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2e34044873724ad79d7fcae000cf643d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680288987017 2023-03-31 18:56:28,701 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting b115b79ab175439687681515cceacf31, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680288987017 2023-03-31 18:56:28,714 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:28,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:28,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:28,717 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#B#compaction#564 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:28,750 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#A#compaction#563 average throughput is 0.56 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:28,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:28,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:28,751 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:28,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:28,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:28,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:28,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,751 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,753 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:28,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:28,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,753 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:28,756 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331ecf8092d4c8546979e36ace133521f99 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ecf8092d4c8546979e36ace133521f99 2023-03-31 18:56:28,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:28,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289048756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:28,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:28,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:28,775 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:28,775 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:28,775 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:28,775 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:28,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:28,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:28,775 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,776 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:28,776 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:28,777 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d13ecb7[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:28,780 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:28,780 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,781 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:28,781 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,781 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:28,781 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/df000f06b43a44b8bb3ffa75acb7f264 2023-03-31 18:56:28,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:28,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:28,784 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,784 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:28,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:28,788 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:28,788 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:28,788 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:28,788 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:28,788 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:28,789 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,789 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,789 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:28,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:28,790 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,791 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,793 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:28,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:28,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:28,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:28,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:28,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:28,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:28,793 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:28,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,794 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,795 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:28,795 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:28,818 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/73c5ef4e8be740f9bf929200d02581d4 2023-03-31 18:56:28,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:28,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289048859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,895 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:28,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:28,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:28,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:28,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:28,898 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:28,898 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:28,898 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:28,898 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:28,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,899 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,900 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:28,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,901 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,902 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:28,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:28,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:28,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:28,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:28,903 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:28,903 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:28,903 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:28,903 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:28,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:28,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:28,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:28,903 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:28,905 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:28,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:28,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:28,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:28,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:28,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:28,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645612c9[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:28,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:28,908 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,912 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,913 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:28,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,913 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:28,913 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:28,913 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:28,915 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,915 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:28,915 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:28,915 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:28,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:28,917 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:28,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:28,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:28,917 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:28,918 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:28,919 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,919 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:28,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,919 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:28,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:28,920 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,921 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,923 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:28,923 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:28,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:28,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:28,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:28,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:28,923 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,924 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:28,924 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:28,923 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:28,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:28,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:28,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:28,924 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:28,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:29,024 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:29,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:29,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:29,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:29,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:29,028 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:29,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:29,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:29,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:29,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,033 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,036 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:29,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:29,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,036 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:29,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:29,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:29,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:29,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:29,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:29,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:29,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:29,038 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:29,038 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:29,038 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:29,038 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:29,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:29,038 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:29,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:29,039 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:29,040 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,040 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:29,040 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa8fb32[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:29,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:29,042 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,043 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:29,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:29,044 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:29,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:29,046 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:29,046 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:29,046 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,046 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:29,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:29,048 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:29,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:29,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:29,048 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:29,048 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:29,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:29,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:29,051 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:29,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:29,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:29,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:29,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,052 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:29,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,054 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,056 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:29,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:29,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:29,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:29,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:29,057 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,057 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:29,057 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:29,057 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:29,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:29,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:29,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289049064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:29,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289049147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,156 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/e05bf0a0ea914454b7e3a714639d88e1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e05bf0a0ea914454b7e3a714639d88e1 2023-03-31 18:56:29,157 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:29,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:29,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:29,160 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/B of f34b89af9bae1a93ef5b2acb275cb610 into e05bf0a0ea914454b7e3a714639d88e1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:29,160 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:29,160 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/B, priority=11, startTime=1680288988696; duration=0sec 2023-03-31 18:56:29,161 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:29,161 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 18:56:29,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:29,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:29,161 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:29,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:29,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:29,161 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:29,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,162 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,162 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72088 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 18:56:29,163 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/C is initiating minor compaction (all files) 2023-03-31 18:56:29,163 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/C in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:29,163 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:29,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:29,163 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/cb156128cd3c4bea808718aa8c63efb6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3110c06252a14e8aa9b68f26bd5060b7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3ec5ca73b3284d9eb23e48e03b2693bd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c6ff96acc3ab4838b57e2dba6abb879b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d7067f38ba654fa690b4ec6200d16112, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/7cf4e3090cbf4beaba15bdc46b9a9bf1] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=70.4 K 2023-03-31 18:56:29,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,163 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,163 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting cb156128cd3c4bea808718aa8c63efb6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680288982496 2023-03-31 18:56:29,164 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3110c06252a14e8aa9b68f26bd5060b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680288984371 2023-03-31 18:56:29,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,164 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 3ec5ca73b3284d9eb23e48e03b2693bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1680288984497 2023-03-31 18:56:29,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:29,165 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting c6ff96acc3ab4838b57e2dba6abb879b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680288985123 2023-03-31 18:56:29,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:29,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:29,165 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d7067f38ba654fa690b4ec6200d16112, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680288986343 2023-03-31 18:56:29,165 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:29,166 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:29,166 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 7cf4e3090cbf4beaba15bdc46b9a9bf1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680288987017 2023-03-31 18:56:29,166 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:29,166 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:29,167 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733a2b00[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:29,169 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:29,169 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,172 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:29,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:29,172 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:29,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,172 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:29,173 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:29,173 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,175 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#C#compaction#567 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:29,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:29,176 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:29,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:29,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:29,177 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:29,177 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:29,178 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,178 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:29,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,178 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:29,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,179 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,181 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,182 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:29,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:29,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:29,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:29,183 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:29,183 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:29,183 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,183 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:29,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:29,183 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,183 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:29,183 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,183 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:29,183 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/fcb83efc22914babbc938e3874953edd 2023-03-31 18:56:29,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/df000f06b43a44b8bb3ffa75acb7f264 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df000f06b43a44b8bb3ffa75acb7f264 2023-03-31 18:56:29,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df000f06b43a44b8bb3ffa75acb7f264, entries=150, sequenceid=240, filesize=25.3 K 2023-03-31 18:56:29,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/73c5ef4e8be740f9bf929200d02581d4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73c5ef4e8be740f9bf929200d02581d4 2023-03-31 18:56:29,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73c5ef4e8be740f9bf929200d02581d4, entries=150, sequenceid=240, filesize=11.8 K 2023-03-31 18:56:29,241 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/fcb83efc22914babbc938e3874953edd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/fcb83efc22914babbc938e3874953edd 2023-03-31 18:56:29,245 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/fcb83efc22914babbc938e3874953edd, entries=150, sequenceid=240, filesize=11.8 K 2023-03-31 18:56:29,245 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for f34b89af9bae1a93ef5b2acb275cb610 in 546ms, sequenceid=240, compaction requested=false 2023-03-31 18:56:29,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:29,283 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:29,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:29,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:29,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:29,287 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:29,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:29,287 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:29,288 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:29,288 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,288 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,290 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:29,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:29,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,290 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:29,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:29,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:29,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:29,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:29,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:29,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:29,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:29,292 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:29,292 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:29,292 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:29,293 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:56:29,293 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:29,293 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:29,293 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:29,293 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:29,293 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:29,293 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:29,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:29,372 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:29,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:29,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289049449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:29,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289049554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:29,584 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/9034b3ac7e6e49089b7b40822d004865 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9034b3ac7e6e49089b7b40822d004865 2023-03-31 18:56:29,587 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/C of f34b89af9bae1a93ef5b2acb275cb610 into 9034b3ac7e6e49089b7b40822d004865(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:29,587 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:29,587 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/C, priority=10, startTime=1680288988696; duration=0sec 2023-03-31 18:56:29,587 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:29,615 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/0e0924523bc54205b1a1707bed5d05f8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/0e0924523bc54205b1a1707bed5d05f8 2023-03-31 18:56:29,619 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/A of f34b89af9bae1a93ef5b2acb275cb610 into 0e0924523bc54205b1a1707bed5d05f8(size=25.7 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:29,619 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:29,619 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/A, priority=11, startTime=1680288988696; duration=0sec 2023-03-31 18:56:29,619 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:29,717 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331d11a399e9ab34a1d96f9c2c74762804a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d11a399e9ab34a1d96f9c2c74762804a 2023-03-31 18:56:29,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:29,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289049758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289050060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,123 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=253, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/4d8350e9d6c84853adf4e770dfcef492 2023-03-31 18:56:30,535 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/a6a06d81c12a49c4aa3088b238a41efd 2023-03-31 18:56:30,546 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/d61671cd4ecb4d42b815caa3fa575150 2023-03-31 18:56:30,550 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/4d8350e9d6c84853adf4e770dfcef492 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4d8350e9d6c84853adf4e770dfcef492 2023-03-31 18:56:30,553 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4d8350e9d6c84853adf4e770dfcef492, entries=150, sequenceid=253, filesize=25.3 K 2023-03-31 18:56:30,554 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/a6a06d81c12a49c4aa3088b238a41efd as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a6a06d81c12a49c4aa3088b238a41efd 2023-03-31 18:56:30,557 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a6a06d81c12a49c4aa3088b238a41efd, entries=150, sequenceid=253, filesize=11.8 K 2023-03-31 18:56:30,558 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/d61671cd4ecb4d42b815caa3fa575150 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d61671cd4ecb4d42b815caa3fa575150 2023-03-31 18:56:30,561 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d61671cd4ecb4d42b815caa3fa575150, entries=150, sequenceid=253, filesize=11.8 K 2023-03-31 18:56:30,561 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f34b89af9bae1a93ef5b2acb275cb610 in 1268ms, sequenceid=253, compaction requested=true 2023-03-31 18:56:30,562 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:30,562 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:30,562 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:30,562 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:30,562 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,562 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:30,562 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:30,563 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,564 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,564 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,564 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:30,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61de0784[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:30,565 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:30,566 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:30,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-31 18:56:30,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:30,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:30,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:30,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:30,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:30,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:30,568 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:30,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:30,569 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:30,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,569 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:30,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:30,574 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:30,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:30,578 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:30,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:30,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:30,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:30,578 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:30,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,581 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:30,581 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,583 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033195ea9a6471f647e8ab516d4e568921e2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033195ea9a6471f647e8ab516d4e568921e2 2023-03-31 18:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,583 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:30,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:30,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:30,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,588 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:30,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:30,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:30,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:30,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,588 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:30,588 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:30,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,588 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,590 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=276, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/eb752248766440f2801eb41badc5972d 2023-03-31 18:56:30,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/43aee40f4cf34d6388ae2d0d7c229159 2023-03-31 18:56:30,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289050603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/bbd18f62bc894274b6061e25c11c34e5 2023-03-31 18:56:30,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/eb752248766440f2801eb41badc5972d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eb752248766440f2801eb41badc5972d 2023-03-31 18:56:30,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eb752248766440f2801eb41badc5972d, entries=150, sequenceid=276, filesize=25.4 K 2023-03-31 18:56:30,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/43aee40f4cf34d6388ae2d0d7c229159 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/43aee40f4cf34d6388ae2d0d7c229159 2023-03-31 18:56:30,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/43aee40f4cf34d6388ae2d0d7c229159, entries=150, sequenceid=276, filesize=11.9 K 2023-03-31 18:56:30,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/bbd18f62bc894274b6061e25c11c34e5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bbd18f62bc894274b6061e25c11c34e5 2023-03-31 18:56:30,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bbd18f62bc894274b6061e25c11c34e5, entries=150, sequenceid=276, filesize=11.9 K 2023-03-31 18:56:30,625 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for f34b89af9bae1a93ef5b2acb275cb610 in 58ms, sequenceid=276, compaction requested=true 2023-03-31 18:56:30,625 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:30,625 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:30,625 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:30,625 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:30,625 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:30,625 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:30,626 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104144 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:30,626 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48704 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:30,626 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/A is initiating minor compaction (all files) 2023-03-31 18:56:30,626 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/B is initiating minor compaction (all files) 2023-03-31 18:56:30,626 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/A in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:30,626 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/B in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:30,626 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/0e0924523bc54205b1a1707bed5d05f8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df000f06b43a44b8bb3ffa75acb7f264, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4d8350e9d6c84853adf4e770dfcef492, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eb752248766440f2801eb41badc5972d] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=101.7 K 2023-03-31 18:56:30,626 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e05bf0a0ea914454b7e3a714639d88e1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73c5ef4e8be740f9bf929200d02581d4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a6a06d81c12a49c4aa3088b238a41efd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/43aee40f4cf34d6388ae2d0d7c229159] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=47.6 K 2023-03-31 18:56:30,627 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 0e0924523bc54205b1a1707bed5d05f8, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680288987017 2023-03-31 18:56:30,627 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e05bf0a0ea914454b7e3a714639d88e1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680288987017 2023-03-31 18:56:30,627 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting df000f06b43a44b8bb3ffa75acb7f264, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680288988689 2023-03-31 18:56:30,627 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 73c5ef4e8be740f9bf929200d02581d4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680288988689 2023-03-31 18:56:30,628 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 4d8350e9d6c84853adf4e770dfcef492, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680288988714 2023-03-31 18:56:30,628 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a6a06d81c12a49c4aa3088b238a41efd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680288988714 2023-03-31 18:56:30,628 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting eb752248766440f2801eb41badc5972d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680288989426 2023-03-31 18:56:30,628 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 43aee40f4cf34d6388ae2d0d7c229159, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680288989426 2023-03-31 18:56:30,638 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#B#compaction#574 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:30,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:30,640 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:56:30,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:30,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:30,641 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:30,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:30,641 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:30,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:30,643 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#A#compaction#575 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:30,655 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/9456146bac7e4d3ebc350ad617442780 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9456146bac7e4d3ebc350ad617442780 2023-03-31 18:56:30,660 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/B of f34b89af9bae1a93ef5b2acb275cb610 into 9456146bac7e4d3ebc350ad617442780(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:30,660 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:30,660 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/B, priority=12, startTime=1680288990625; duration=0sec 2023-03-31 18:56:30,660 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:30,660 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:30,662 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48670 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:30,662 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/C is initiating minor compaction (all files) 2023-03-31 18:56:30,662 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/C in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:30,662 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9034b3ac7e6e49089b7b40822d004865, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/fcb83efc22914babbc938e3874953edd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d61671cd4ecb4d42b815caa3fa575150, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bbd18f62bc894274b6061e25c11c34e5] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=47.5 K 2023-03-31 18:56:30,662 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9034b3ac7e6e49089b7b40822d004865, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680288987017 2023-03-31 18:56:30,663 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting fcb83efc22914babbc938e3874953edd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680288988689 2023-03-31 18:56:30,663 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d61671cd4ecb4d42b815caa3fa575150, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680288988714 2023-03-31 18:56:30,663 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting bbd18f62bc894274b6061e25c11c34e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680288989426 2023-03-31 18:56:30,665 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331d0960788d9544d3391f7469f2af8fd8e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d0960788d9544d3391f7469f2af8fd8e 2023-03-31 18:56:30,670 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=288, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/65975a42fb2a4d90a179aa70283699e1 2023-03-31 18:56:30,671 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#C#compaction#577 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:30,687 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/d492fd3079064ef6959d791b33c02233 2023-03-31 18:56:30,688 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/bba7259516eb46cc9198ee63135d6da9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bba7259516eb46cc9198ee63135d6da9 2023-03-31 18:56:30,688 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:30,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:30,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:30,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289050688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33012 deadline: 1680289050689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,694 DEBUG [Thread-3973] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:56:30,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:30,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:30,695 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:30,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:30,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:30,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:30,696 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,696 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:30,697 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:30,697 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,698 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289050693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:30,702 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/C of f34b89af9bae1a93ef5b2acb275cb610 into bba7259516eb46cc9198ee63135d6da9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:30,702 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:30,702 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/C, priority=12, startTime=1680288990625; duration=0sec 2023-03-31 18:56:30,702 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:30,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:30,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:30,709 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:30,710 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:30,710 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:30,710 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:30,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289050707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,712 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ca812c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:30,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:30,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,715 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,715 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:30,716 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:30,716 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:30,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:30,717 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:30,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:30,720 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:30,720 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:30,720 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:30,720 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:30,720 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:30,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:30,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,723 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,724 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/20f10031814e4404ad1bb535dfd86f37 2023-03-31 18:56:30,734 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:30,734 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:30,734 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:30,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:30,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:30,734 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,734 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,734 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,734 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,734 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,734 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,734 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:30,734 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:30,734 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:30,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:30,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/65975a42fb2a4d90a179aa70283699e1 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/65975a42fb2a4d90a179aa70283699e1 2023-03-31 18:56:30,739 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/65975a42fb2a4d90a179aa70283699e1, entries=200, sequenceid=288, filesize=32.3 K 2023-03-31 18:56:30,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/d492fd3079064ef6959d791b33c02233 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/d492fd3079064ef6959d791b33c02233 2023-03-31 18:56:30,742 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/d492fd3079064ef6959d791b33c02233, entries=150, sequenceid=288, filesize=11.9 K 2023-03-31 18:56:30,743 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/20f10031814e4404ad1bb535dfd86f37 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/20f10031814e4404ad1bb535dfd86f37 2023-03-31 18:56:30,746 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/20f10031814e4404ad1bb535dfd86f37, entries=150, sequenceid=288, filesize=11.9 K 2023-03-31 18:56:30,746 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f34b89af9bae1a93ef5b2acb275cb610 in 106ms, sequenceid=288, compaction requested=false 2023-03-31 18:56:30,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:30,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:30,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-31 18:56:30,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:30,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:30,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:30,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:30,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:30,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:30,808 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033135295f5805a24b088ad0f8b81a84f18a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033135295f5805a24b088ad0f8b81a84f18a 2023-03-31 18:56:30,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289050812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289050815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,822 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/3e77f73cbcbb4cb09eca4c0a05892a32 2023-03-31 18:56:30,834 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:30,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:30,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:30,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:30,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:30,838 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:30,838 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:30,838 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:30,838 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:30,839 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,839 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,840 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:30,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:30,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:30,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:30,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:30,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:30,842 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:30,842 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:30,842 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:30,842 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:30,844 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,844 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,845 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:30,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d339dcc[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:30,846 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:30,846 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:30,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:30,849 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,849 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:30,849 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,849 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:30,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:30,851 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:30,852 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:30,852 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:30,852 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:30,852 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:30,853 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,853 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,853 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:30,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:30,854 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,855 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,858 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,858 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,858 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:30,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:30,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:30,858 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:30,858 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,858 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,858 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,858 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,858 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:30,858 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:30,858 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:30,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:30,858 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:30,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:30,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:30,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289050912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289050916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:30,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289050919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,958 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:30,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:30,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:30,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:30,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:30,962 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:30,962 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:30,962 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:30,962 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:30,962 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,962 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,964 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:30,964 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:30,964 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,964 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:30,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:30,966 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:30,966 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:30,966 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:30,966 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:30,968 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:30,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff3c341[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:30,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:30,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,971 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:30,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:30,971 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:30,971 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,971 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:30,974 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:30,974 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:30,974 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,974 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:30,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:30,976 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:30,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:30,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:30,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:30,976 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:30,977 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:30,978 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:30,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,978 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:30,979 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:30,984 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:30,984 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,984 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,984 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:30,984 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:30,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:30,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:30,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:30,984 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:30,985 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:30,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:30,985 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:30,985 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:30,985 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,985 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:30,985 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:30,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:31,085 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:31,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:31,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:31,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:31,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:31,089 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:31,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:31,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:31,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:31,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,089 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,091 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:31,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:31,091 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,092 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,092 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:31,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:31,093 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:31,093 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:31,094 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:31,094 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:31,096 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,096 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:31,096 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,096 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:31,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:31,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e00051c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:31,098 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:31,098 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,099 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:31,099 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,099 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:31,101 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:31,101 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:31,101 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:31,101 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:31,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:31,103 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:31,103 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:31,103 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:31,103 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:31,103 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:31,106 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,106 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,106 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:31,106 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:31,108 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,110 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:31,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:31,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:31,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:31,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,111 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:31,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:31,111 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:31,111 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:31,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:31,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289051119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289051122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289051156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,161 DEBUG [Thread-3979] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:56:31,211 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:31,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:31,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:31,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:31,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:31,214 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:31,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:31,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:31,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:31,215 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,215 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,216 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:31,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:31,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:31,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:31,218 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:31,218 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:31,218 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:31,218 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:31,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289051217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:31,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:31,220 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:31,220 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,220 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,220 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa22611[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:31,222 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:31,222 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,225 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:31,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:31,225 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:31,225 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:31,226 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,226 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:31,226 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:31,226 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:31,228 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:31,228 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:31,229 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:31,229 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:31,229 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:31,230 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,230 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:31,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,230 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:31,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:31,231 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/92caf5deb6444d839c3d4df34318a318 2023-03-31 18:56:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,233 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,236 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,236 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:31,236 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:31,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:31,236 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,236 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:31,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:31,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:31,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:31,236 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:31,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,237 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:31,237 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:31,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:31,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:31,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/013bb6814cb149dc85e58579ed86866d 2023-03-31 18:56:31,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/3e77f73cbcbb4cb09eca4c0a05892a32 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/3e77f73cbcbb4cb09eca4c0a05892a32 2023-03-31 18:56:31,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/3e77f73cbcbb4cb09eca4c0a05892a32, entries=200, sequenceid=311, filesize=32.3 K 2023-03-31 18:56:31,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/92caf5deb6444d839c3d4df34318a318 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/92caf5deb6444d839c3d4df34318a318 2023-03-31 18:56:31,252 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/92caf5deb6444d839c3d4df34318a318, entries=150, sequenceid=311, filesize=11.9 K 2023-03-31 18:56:31,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/013bb6814cb149dc85e58579ed86866d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/013bb6814cb149dc85e58579ed86866d 2023-03-31 18:56:31,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/013bb6814cb149dc85e58579ed86866d, entries=150, sequenceid=311, filesize=11.9 K 2023-03-31 18:56:31,256 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for f34b89af9bae1a93ef5b2acb275cb610 in 461ms, sequenceid=311, compaction requested=true 2023-03-31 18:56:31,256 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:31,257 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:31,257 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:31,257 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 18:56:31,257 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-03-31 18:56:31,257 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 18:56:31,257 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 18:56:31,257 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. because compaction request was cancelled 2023-03-31 18:56:31,257 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:31,258 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:31,258 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/B is initiating minor compaction (all files) 2023-03-31 18:56:31,258 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/B in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:31,258 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9456146bac7e4d3ebc350ad617442780, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/d492fd3079064ef6959d791b33c02233, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/92caf5deb6444d839c3d4df34318a318] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=36.2 K 2023-03-31 18:56:31,259 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9456146bac7e4d3ebc350ad617442780, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680288989426 2023-03-31 18:56:31,259 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d492fd3079064ef6959d791b33c02233, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680288990582 2023-03-31 18:56:31,259 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 92caf5deb6444d839c3d4df34318a318, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680288990686 2023-03-31 18:56:31,264 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#B#compaction#583 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:31,271 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/2a32d880f2fa4a22a41208cfafe3e78e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2a32d880f2fa4a22a41208cfafe3e78e 2023-03-31 18:56:31,275 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/B of f34b89af9bae1a93ef5b2acb275cb610 into 2a32d880f2fa4a22a41208cfafe3e78e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:31,275 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:31,275 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/B, priority=13, startTime=1680288991257; duration=0sec 2023-03-31 18:56:31,275 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:31,275 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:31,275 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:31,275 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/C is initiating minor compaction (all files) 2023-03-31 18:56:31,275 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/C in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:31,276 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bba7259516eb46cc9198ee63135d6da9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/20f10031814e4404ad1bb535dfd86f37, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/013bb6814cb149dc85e58579ed86866d] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=36.2 K 2023-03-31 18:56:31,276 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting bba7259516eb46cc9198ee63135d6da9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680288989426 2023-03-31 18:56:31,276 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 20f10031814e4404ad1bb535dfd86f37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680288990582 2023-03-31 18:56:31,277 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 013bb6814cb149dc85e58579ed86866d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680288990686 2023-03-31 18:56:31,281 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#C#compaction#584 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:31,289 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/9545295a115e4cebb4840f6b50f2db82 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9545295a115e4cebb4840f6b50f2db82 2023-03-31 18:56:31,293 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/C of f34b89af9bae1a93ef5b2acb275cb610 into 9545295a115e4cebb4840f6b50f2db82(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:31,293 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:31,293 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/C, priority=13, startTime=1680288991257; duration=0sec 2023-03-31 18:56:31,293 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:31,337 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:31,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:31,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:31,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:31,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:31,340 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:31,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:31,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:31,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:31,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,341 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,343 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,343 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:31,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:31,343 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,343 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:31,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:31,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:31,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:31,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:31,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:31,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:31,345 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:31,345 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:31,345 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:56:31,346 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:31,346 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:31,346 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:31,346 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:31,346 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:31,346 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:31,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:31,426 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:31,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288991448, max timestamp=1680288991455Num uniques 859; 2023-03-31 18:56:31,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:31,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288991448, max timestamp=1680288991455Num uniques 859; 2023-03-31 18:56:31,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:31,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288991448, max timestamp=1680288991455Num uniques 859; 2023-03-31 18:56:31,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:31,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289051483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289051485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289051587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289051588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289051723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,759 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033100ddb7876a9c446ab3a1b12b2e2f71fa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033100ddb7876a9c446ab3a1b12b2e2f71fa 2023-03-31 18:56:31,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289051790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:31,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289051790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:31,868 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/c323571baf534a54b4530cfcb180c1c5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c323571baf534a54b4530cfcb180c1c5 2023-03-31 18:56:31,871 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/A of f34b89af9bae1a93ef5b2acb275cb610 into c323571baf534a54b4530cfcb180c1c5(size=26.0 K), total size for store is 90.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-31 18:56:31,872 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:31,872 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/A, priority=12, startTime=1680288990625; duration=1sec 2023-03-31 18:56:31,872 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:32,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:32,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289052095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:32,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:32,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289052096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:32,164 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=324, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/c23b5c4d0283411580327ec6289e09cb 2023-03-31 18:56:32,578 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/607dbd5229e748908c3ab73bcd1218a9 2023-03-31 18:56:32,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:32,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289052600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:32,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:32,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289052601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:32,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:32,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289052734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:32,987 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/8e10d8e674b343099813a6bd10c79d0c 2023-03-31 18:56:32,991 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/c23b5c4d0283411580327ec6289e09cb as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c23b5c4d0283411580327ec6289e09cb 2023-03-31 18:56:32,994 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c23b5c4d0283411580327ec6289e09cb, entries=150, sequenceid=324, filesize=25.4 K 2023-03-31 18:56:32,995 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/607dbd5229e748908c3ab73bcd1218a9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/607dbd5229e748908c3ab73bcd1218a9 2023-03-31 18:56:33,003 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/607dbd5229e748908c3ab73bcd1218a9, entries=150, sequenceid=324, filesize=11.9 K 2023-03-31 18:56:33,004 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/8e10d8e674b343099813a6bd10c79d0c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8e10d8e674b343099813a6bd10c79d0c 2023-03-31 18:56:33,014 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8e10d8e674b343099813a6bd10c79d0c, entries=150, sequenceid=324, filesize=11.9 K 2023-03-31 18:56:33,014 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f34b89af9bae1a93ef5b2acb275cb610 in 1669ms, sequenceid=324, compaction requested=true 2023-03-31 18:56:33,014 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:33,014 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:33,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:33,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:33,015 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:33,016 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,016 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,017 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e5e0bd4[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:33,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:33,019 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,020 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:33,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,020 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:33,021 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,021 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:33,021 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,022 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:33,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:33,025 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:33,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:33,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:33,025 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:33,025 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:33,027 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,027 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,027 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,028 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,030 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,034 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:33,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:33,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,035 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,035 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:33,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,035 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:33,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,035 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,135 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:33,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:33,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:33,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:33,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:33,139 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:33,139 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,139 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:33,139 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:33,139 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,139 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,141 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,141 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,141 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,141 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:33,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:33,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:33,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:33,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:33,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:33,142 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:33,142 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,142 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:33,142 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-31 18:56:33,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:33,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:33,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:33,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:33,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:33,143 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:33,155 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033129c6579d2cf1483587a44e30dc2edfe5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033129c6579d2cf1483587a44e30dc2edfe5 2023-03-31 18:56:33,160 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=361, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/7d37f70ee0ff4873a213a04ee04f4eee 2023-03-31 18:56:33,170 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/c37e93b1d4d24444a52d42774e5f8672 2023-03-31 18:56:33,179 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/d79719dc291a453786c67f4015df6fb2 2023-03-31 18:56:33,183 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/7d37f70ee0ff4873a213a04ee04f4eee as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/7d37f70ee0ff4873a213a04ee04f4eee 2023-03-31 18:56:33,186 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/7d37f70ee0ff4873a213a04ee04f4eee, entries=150, sequenceid=361, filesize=25.4 K 2023-03-31 18:56:33,187 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/c37e93b1d4d24444a52d42774e5f8672 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/c37e93b1d4d24444a52d42774e5f8672 2023-03-31 18:56:33,190 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/c37e93b1d4d24444a52d42774e5f8672, entries=150, sequenceid=361, filesize=11.9 K 2023-03-31 18:56:33,191 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/d79719dc291a453786c67f4015df6fb2 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d79719dc291a453786c67f4015df6fb2 2023-03-31 18:56:33,194 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d79719dc291a453786c67f4015df6fb2, entries=150, sequenceid=361, filesize=11.9 K 2023-03-31 18:56:33,194 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for f34b89af9bae1a93ef5b2acb275cb610 in 52ms, sequenceid=361, compaction requested=true 2023-03-31 18:56:33,194 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:33,195 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:33,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:33,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:33,195 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:33,198 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,198 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,198 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,198 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:33,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f362095[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:33,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:33,200 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,201 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,201 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:33,201 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:33,201 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:33,201 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,201 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:33,203 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:33,203 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,203 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,203 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:33,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:33,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:33,205 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:33,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:33,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:33,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:33,205 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:33,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,208 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,209 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,212 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:33,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:33,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:33,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,213 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:33,213 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:33,314 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:33,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:33,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:33,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:33,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:33,317 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:33,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:33,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:33,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,319 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,319 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,320 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:33,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:33,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:33,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:33,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:33,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:33,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:33,321 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,321 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:33,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:33,322 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:33,322 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,322 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:33,322 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:33,322 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,322 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:33,322 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:33,323 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,324 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,324 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:33,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:33,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4110f48a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:33,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:33,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,326 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:33,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:33,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:33,327 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,327 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:33,329 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:33,329 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,329 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,329 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:33,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:33,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:33,332 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:33,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:33,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:33,332 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:33,332 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:33,334 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,334 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,334 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,334 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,336 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,338 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:33,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:33,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,339 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:33,339 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:33,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:33,439 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:33,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:33,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:33,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:33,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:33,442 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:33,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:33,442 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:33,443 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,443 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,444 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:33,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:33,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:33,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:33,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:33,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:33,446 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:33,446 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:33,446 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:33,446 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:33,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:33,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:33,446 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:33,448 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:33,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:33,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ffcd017[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:33,450 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:33,450 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,452 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,452 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:33,452 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:33,452 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:33,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:33,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:33,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,453 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:33,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:33,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:33,456 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:33,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:33,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:33,456 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:33,456 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:33,458 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,458 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,458 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,460 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:33,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:33,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,463 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:33,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,463 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:33,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,464 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,464 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,464 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,564 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:33,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:33,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:33,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:33,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:33,567 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:33,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:33,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:33,567 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,568 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,572 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,572 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,572 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,572 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:33,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:33,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:33,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:33,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:33,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:33,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,573 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:33,574 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,574 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:33,574 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:33,574 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:33,576 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:33,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1426c97[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:33,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:33,578 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,580 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:33,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:33,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:33,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:33,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,582 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:33,582 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,582 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:33,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:33,584 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:33,584 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:33,584 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:33,584 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:33,584 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:33,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,586 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,587 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,588 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,591 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,591 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,591 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:33,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:33,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:33,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,591 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:33,591 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:33,591 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,591 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,591 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,591 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,591 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:33,643 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:56:33,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:33,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:33,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:33,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:33,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:33,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:33,676 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303312ba3e94992c54f79b4ccd90ccb785dfa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312ba3e94992c54f79b4ccd90ccb785dfa 2023-03-31 18:56:33,681 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=375, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/91016faf983d44da81803cf1eb3ca2ff 2023-03-31 18:56:33,691 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:33,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:33,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:33,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:33,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:33,695 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:33,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:33,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:33,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,695 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,697 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,697 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,698 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,698 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:33,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:33,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289053696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288993691, max timestamp=1680288993698Num uniques 859; 2023-03-31 18:56:33,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:33,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:33,700 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:33,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288993691, max timestamp=1680288993698Num uniques 859; 2023-03-31 18:56:33,700 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:33,700 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:33,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288993691, max timestamp=1680288993698Num uniques 859; 2023-03-31 18:56:33,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:33,701 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:33,701 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:33,701 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:33,701 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:33,702 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:33,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f0ed6a2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:33,706 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:33,706 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,707 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:33,707 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:33,707 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:33,708 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:33,709 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,709 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,709 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:33,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:33,712 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:33,712 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:33,712 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:33,712 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:33,712 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:33,718 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,718 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,718 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,718 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,719 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,721 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,721 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,721 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:33,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:33,721 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,721 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,722 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:33,722 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:33,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:33,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289053766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:33,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289053802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,822 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:33,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:33,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:33,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:33,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:33,825 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:33,825 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:33,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:33,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,827 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,827 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:33,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:33,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:33,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:33,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:33,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:33,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:33,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:33,829 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,829 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:33,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:33,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:33,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:33,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:33,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:33,831 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,831 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39fcc96d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:33,833 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:33,833 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:33,836 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,836 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:33,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:33,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,838 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:33,840 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:33,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:33,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:33,840 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:33,840 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:33,842 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,842 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,842 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,843 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,844 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:33,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:33,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:33,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,847 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:33,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,847 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:33,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:33,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289053869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,948 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:33,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:33,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:33,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:33,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:33,951 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:33,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:33,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:33,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,951 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,953 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,953 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:33,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:33,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:33,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:33,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:33,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:33,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:33,955 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,955 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:33,956 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:33,956 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:33,956 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:33,956 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:33,956 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,956 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:33,956 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:33,957 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,957 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d9da2d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:33,959 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:33,959 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,964 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:33,964 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,964 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:33,966 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:33,967 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,967 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:33,970 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:33,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:33,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:33,970 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:33,970 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:33,971 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:33,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,972 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:33,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,973 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,976 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,976 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:33,976 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,976 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:33,976 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:33,976 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,976 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:33,976 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:33,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:33,976 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:33,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:33,976 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:33,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:33,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:33,980 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:33,980 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:33,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:34,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:34,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289054005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:34,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289054071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,080 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:34,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:34,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:34,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:34,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:34,083 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:34,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:34,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:34,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:34,084 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,084 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,085 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:34,085 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:34,086 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,086 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:34,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:34,087 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:34,087 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:34,087 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:34,087 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:34,087 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,087 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:34,087 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:34,088 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:34,088 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:34,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:34,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:34,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:34,091 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,091 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,091 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,091 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/9e14fc7cfd3d42229f9d6208abd22c8c 2023-03-31 18:56:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ed2475[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:34,093 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:34,094 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,095 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,095 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:34,096 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,096 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:34,098 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:34,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:34,101 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:34,101 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:34,101 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:34,101 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:34,101 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:34,103 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,103 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:34,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,103 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,104 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,106 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,109 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:34,109 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:34,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:34,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:34,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:34,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,110 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:34,110 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:34,109 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:34,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:34,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,110 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,111 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,210 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:34,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:34,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:34,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:34,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:34,213 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:34,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:34,213 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:34,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:34,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,214 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,215 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,215 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:34,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:34,215 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:34,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:34,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:34,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:34,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:34,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:34,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:34,217 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,217 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:34,217 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:34,217 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:34,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:34,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:34,218 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:34,219 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:34,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b5f574c[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:34,221 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:34,221 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,222 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,222 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,222 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:34,222 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:34,222 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:34,222 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:34,225 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,225 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,225 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:34,225 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:34,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:34,228 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:34,228 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:34,228 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:34,228 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:34,228 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:34,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,229 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:34,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:34,230 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,231 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,233 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,233 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:34,233 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,233 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:34,233 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:34,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:34,233 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:34,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:34,233 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,234 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,234 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,234 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:34,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:34,234 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:34,234 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:34,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:34,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289054309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,334 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:34,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:34,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:34,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:34,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:34,338 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:34,338 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:34,338 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:34,338 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:34,338 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,338 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:34,339 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:34,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,340 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:34,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:34,341 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:34,341 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:34,341 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:34,341 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:34,341 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:34,342 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,342 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:34,342 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:34,342 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:34,344 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,344 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24374eeb[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:34,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:34,346 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,348 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:34,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:34,352 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,352 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:34,352 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,352 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:34,355 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:34,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:34,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:34,355 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:34,355 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:34,359 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,359 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,359 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:34,360 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,361 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:34,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:34,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:34,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:34,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:34,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:34,365 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:34,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,365 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:34,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,365 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:34,365 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:34,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:34,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33016 deadline: 1680289054375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,466 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:34,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:34,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:34,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:34,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:34,470 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:34,470 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:34,470 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:34,470 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:34,470 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,470 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,472 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:34,472 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:34,472 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,472 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:34,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:34,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:34,474 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,474 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:34,474 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:34,474 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:34,477 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,477 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,477 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,477 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:34,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a91e0f2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:34,479 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:34,479 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,481 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:34,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:34,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,481 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:34,481 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:34,483 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:34,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:34,486 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:34,486 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:34,486 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:34,486 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:34,486 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:34,488 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,488 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:34,488 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:34,489 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:34,490 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,493 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:34,493 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:34,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:34,493 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:34,493 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:34,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:34,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:34,493 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,493 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,493 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:34,493 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:34,493 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:34,493 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,494 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:34,494 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,503 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/a987f28ad83f47f58006870fd68d09ad 2023-03-31 18:56:34,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/91016faf983d44da81803cf1eb3ca2ff as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/91016faf983d44da81803cf1eb3ca2ff 2023-03-31 18:56:34,511 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/91016faf983d44da81803cf1eb3ca2ff, entries=150, sequenceid=375, filesize=25.4 K 2023-03-31 18:56:34,513 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/9e14fc7cfd3d42229f9d6208abd22c8c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9e14fc7cfd3d42229f9d6208abd22c8c 2023-03-31 18:56:34,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9e14fc7cfd3d42229f9d6208abd22c8c, entries=150, sequenceid=375, filesize=11.9 K 2023-03-31 18:56:34,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/a987f28ad83f47f58006870fd68d09ad as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/a987f28ad83f47f58006870fd68d09ad 2023-03-31 18:56:34,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/a987f28ad83f47f58006870fd68d09ad, entries=150, sequenceid=375, filesize=11.9 K 2023-03-31 18:56:34,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f34b89af9bae1a93ef5b2acb275cb610 in 877ms, sequenceid=375, compaction requested=true 2023-03-31 18:56:34,520 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:34,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:34,520 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 18:56:34,521 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:34,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:34,521 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:34,522 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:34,522 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/B is initiating minor compaction (all files) 2023-03-31 18:56:34,522 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/B in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,522 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2a32d880f2fa4a22a41208cfafe3e78e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/607dbd5229e748908c3ab73bcd1218a9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/c37e93b1d4d24444a52d42774e5f8672, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9e14fc7cfd3d42229f9d6208abd22c8c] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=48.2 K 2023-03-31 18:56:34,523 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2a32d880f2fa4a22a41208cfafe3e78e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680288990686 2023-03-31 18:56:34,523 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 170902 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 18:56:34,523 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/A is initiating minor compaction (all files) 2023-03-31 18:56:34,523 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/A in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,523 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c323571baf534a54b4530cfcb180c1c5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/65975a42fb2a4d90a179aa70283699e1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/3e77f73cbcbb4cb09eca4c0a05892a32, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c23b5c4d0283411580327ec6289e09cb, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/7d37f70ee0ff4873a213a04ee04f4eee, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/91016faf983d44da81803cf1eb3ca2ff] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=166.9 K 2023-03-31 18:56:34,523 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 607dbd5229e748908c3ab73bcd1218a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680288990805 2023-03-31 18:56:34,523 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c323571baf534a54b4530cfcb180c1c5, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680288989426 2023-03-31 18:56:34,523 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting c37e93b1d4d24444a52d42774e5f8672, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680288991472 2023-03-31 18:56:34,524 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 65975a42fb2a4d90a179aa70283699e1, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680288990579 2023-03-31 18:56:34,524 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9e14fc7cfd3d42229f9d6208abd22c8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680288993622 2023-03-31 18:56:34,524 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 3e77f73cbcbb4cb09eca4c0a05892a32, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680288990681 2023-03-31 18:56:34,524 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting c23b5c4d0283411580327ec6289e09cb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680288990805 2023-03-31 18:56:34,525 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 7d37f70ee0ff4873a213a04ee04f4eee, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680288991472 2023-03-31 18:56:34,525 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 91016faf983d44da81803cf1eb3ca2ff, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680288993622 2023-03-31 18:56:34,532 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#B#compaction#594 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:34,541 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#A#compaction#595 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:34,542 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/8d25f0008b834586ba94717d621debda as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/8d25f0008b834586ba94717d621debda 2023-03-31 18:56:34,546 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/B of f34b89af9bae1a93ef5b2acb275cb610 into 8d25f0008b834586ba94717d621debda(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:34,546 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:34,546 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/B, priority=12, startTime=1680288994520; duration=0sec 2023-03-31 18:56:34,546 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:34,546 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 18:56:34,548 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 18:56:34,548 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/C is initiating minor compaction (all files) 2023-03-31 18:56:34,548 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/C in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,548 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9545295a115e4cebb4840f6b50f2db82, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8e10d8e674b343099813a6bd10c79d0c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d79719dc291a453786c67f4015df6fb2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/a987f28ad83f47f58006870fd68d09ad] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=48.2 K 2023-03-31 18:56:34,549 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 9545295a115e4cebb4840f6b50f2db82, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680288990686 2023-03-31 18:56:34,549 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 8e10d8e674b343099813a6bd10c79d0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680288990805 2023-03-31 18:56:34,549 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting d79719dc291a453786c67f4015df6fb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680288991472 2023-03-31 18:56:34,550 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a987f28ad83f47f58006870fd68d09ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680288993622 2023-03-31 18:56:34,558 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#C#compaction#596 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 18:56:34,565 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/e82ee6458a7847b7861e7b4199556689 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/e82ee6458a7847b7861e7b4199556689 2023-03-31 18:56:34,568 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/34cfcff182144f649d140f0ea66df39c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/34cfcff182144f649d140f0ea66df39c 2023-03-31 18:56:34,571 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/A of f34b89af9bae1a93ef5b2acb275cb610 into e82ee6458a7847b7861e7b4199556689(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:34,571 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:34,571 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/A, priority=10, startTime=1680288994520; duration=0sec 2023-03-31 18:56:34,571 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:34,572 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/C of f34b89af9bae1a93ef5b2acb275cb610 into 34cfcff182144f649d140f0ea66df39c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:34,572 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:34,572 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/C, priority=12, startTime=1680288994521; duration=0sec 2023-03-31 18:56:34,572 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:34,593 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:34,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:34,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:34,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:34,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:34,597 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:34,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:34,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:34,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:34,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,599 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,599 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:34,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:34,600 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,600 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:34,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:34,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:34,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:34,601 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:34,601 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:34,601 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-31 18:56:34,602 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:34,602 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:34,602 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:34,602 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:34,602 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:34,602 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:34,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:34,749 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:34,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:34,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289054768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:34,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33008 deadline: 1680289054815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:34,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289054871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:34,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:34,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289054878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,018 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303312b981a12300446cb81ee8c9f029d7fd3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312b981a12300446cb81ee8c9f029d7fd3 2023-03-31 18:56:35,022 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=414, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/063ce5242abc4d88b8f84d7f42ad1c8e 2023-03-31 18:56:35,031 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/67ff4b921b03467abaa9a6a0f64c6947 2023-03-31 18:56:35,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:35,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289055073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:35,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33034 deadline: 1680289055177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,180 DEBUG [Thread-3979] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., hostname=jenkins-hbase4.apache.org,34951,1680288816339, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 18:56:35,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:35,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33020 deadline: 1680289055376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,441 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/701665bcef6b415f86604569d34eddc5 2023-03-31 18:56:35,445 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/063ce5242abc4d88b8f84d7f42ad1c8e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/063ce5242abc4d88b8f84d7f42ad1c8e 2023-03-31 18:56:35,449 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/063ce5242abc4d88b8f84d7f42ad1c8e, entries=150, sequenceid=414, filesize=25.4 K 2023-03-31 18:56:35,449 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/67ff4b921b03467abaa9a6a0f64c6947 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/67ff4b921b03467abaa9a6a0f64c6947 2023-03-31 18:56:35,453 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/67ff4b921b03467abaa9a6a0f64c6947, entries=150, sequenceid=414, filesize=11.9 K 2023-03-31 18:56:35,453 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/701665bcef6b415f86604569d34eddc5 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/701665bcef6b415f86604569d34eddc5 2023-03-31 18:56:35,457 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/701665bcef6b415f86604569d34eddc5, entries=150, sequenceid=414, filesize=11.9 K 2023-03-31 18:56:35,457 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for f34b89af9bae1a93ef5b2acb275cb610 in 856ms, sequenceid=414, compaction requested=false 2023-03-31 18:56:35,457 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:35,457 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:35,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:35,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:35,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,458 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:35,458 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:35,459 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:35,459 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:35,459 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:35,460 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:35,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:35,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:35,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:35,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:35,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:35,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:35,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1144a359[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:35,461 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:35,462 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:35,463 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:35,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:35,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:35,463 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:35,463 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:35,463 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:35,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:35,464 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:35,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:35,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:35,464 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:35,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:35,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:35,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:35,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:35,467 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:35,467 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:35,467 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:35,467 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:35,467 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:35,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:35,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:35,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:35,470 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:35,471 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:35,472 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:35,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:35,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:35,475 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:35,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:35,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:35,475 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:35,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:35,475 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:35,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:35,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:35,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:35,475 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:35,476 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:35,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:35,575 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:35,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:35,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:35,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:35,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:35,580 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:35,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:35,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:35,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:35,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:35,581 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:35,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:35,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:35,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:35,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,582 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:35,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:35,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:35,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:35,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:35,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:35,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:35,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:35,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:35,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:35,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:35,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:35,584 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:35,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:35,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:35,584 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 18:56:35,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:35,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:35,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:35,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:35,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:35,584 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:35,597 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331ce3cb40c3c6f4b7aac183c46ffc0b2ff to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ce3cb40c3c6f4b7aac183c46ffc0b2ff 2023-03-31 18:56:35,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:35,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:35,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288995856, max timestamp=1680288995874Num uniques 859; 2023-03-31 18:56:35,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:35,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288995856, max timestamp=1680288995874Num uniques 859; 2023-03-31 18:56:35,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:35,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680288995856, max timestamp=1680288995874Num uniques 859; 2023-03-31 18:56:35,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:35,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:35,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289055900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:35,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:35,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289055903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:35,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289055904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,004 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=425, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/98ccbe36cfe14e4a8159def08a1319aa 2023-03-31 18:56:36,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289056005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289056008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289056008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,012 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/af4ae60919e24b8d9f88d77f1412e18e 2023-03-31 18:56:36,022 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/735487691bbc4fb1a414b9d9b14a8e5c 2023-03-31 18:56:36,025 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/98ccbe36cfe14e4a8159def08a1319aa as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/98ccbe36cfe14e4a8159def08a1319aa 2023-03-31 18:56:36,027 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/98ccbe36cfe14e4a8159def08a1319aa, entries=150, sequenceid=425, filesize=25.4 K 2023-03-31 18:56:36,028 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/af4ae60919e24b8d9f88d77f1412e18e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/af4ae60919e24b8d9f88d77f1412e18e 2023-03-31 18:56:36,031 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/af4ae60919e24b8d9f88d77f1412e18e, entries=150, sequenceid=425, filesize=11.9 K 2023-03-31 18:56:36,032 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/735487691bbc4fb1a414b9d9b14a8e5c as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/735487691bbc4fb1a414b9d9b14a8e5c 2023-03-31 18:56:36,034 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/735487691bbc4fb1a414b9d9b14a8e5c, entries=150, sequenceid=425, filesize=11.9 K 2023-03-31 18:56:36,035 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f34b89af9bae1a93ef5b2acb275cb610 in 451ms, sequenceid=425, compaction requested=true 2023-03-31 18:56:36,035 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:36,035 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:36,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:36,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:36,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:36,035 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:36,037 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:36,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,037 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1020730a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:36,038 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:36,039 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,040 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:36,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:36,041 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:36,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,041 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:36,042 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,042 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:36,042 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:36,042 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:36,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:36,044 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:36,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:36,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:36,044 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:36,045 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:36,046 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,046 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:36,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,046 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:36,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:36,047 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,049 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:36,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:36,054 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,055 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:36,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:36,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:36,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:36,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:36,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:36,055 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:36,055 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:36,055 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:36,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:36,155 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:36,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:36,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:36,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:36,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:36,158 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:36,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:36,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:36,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:36,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,158 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:36,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,160 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:36,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:36,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,160 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:36,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:36,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:36,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:36,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:36,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:36,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:36,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:36,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:36,161 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:36,161 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:36,162 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:36,162 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-31 18:56:36,162 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:36,162 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:36,162 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:36,162 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:36,162 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:36,162 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:36,176 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331c4fb014554a04ae9ac9c1245e0d66083 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c4fb014554a04ae9ac9c1245e0d66083 2023-03-31 18:56:36,181 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=461, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/fd6bb06aab9e4a468bd709a0eb78ae78 2023-03-31 18:56:36,193 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/ab513c6159ae495e9c1b41577f351ca9 2023-03-31 18:56:36,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:36,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:36,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289056220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289056223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289056223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289056324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289056327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289056327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289056526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289056530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289056531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,603 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/13ce14112d8543ed8536fe39bba80363 2023-03-31 18:56:36,606 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/fd6bb06aab9e4a468bd709a0eb78ae78 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/fd6bb06aab9e4a468bd709a0eb78ae78 2023-03-31 18:56:36,610 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/fd6bb06aab9e4a468bd709a0eb78ae78, entries=150, sequenceid=461, filesize=25.4 K 2023-03-31 18:56:36,610 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/ab513c6159ae495e9c1b41577f351ca9 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ab513c6159ae495e9c1b41577f351ca9 2023-03-31 18:56:36,613 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ab513c6159ae495e9c1b41577f351ca9, entries=150, sequenceid=461, filesize=11.9 K 2023-03-31 18:56:36,614 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/13ce14112d8543ed8536fe39bba80363 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/13ce14112d8543ed8536fe39bba80363 2023-03-31 18:56:36,617 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/13ce14112d8543ed8536fe39bba80363, entries=150, sequenceid=461, filesize=11.9 K 2023-03-31 18:56:36,617 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for f34b89af9bae1a93ef5b2acb275cb610 in 455ms, sequenceid=461, compaction requested=true 2023-03-31 18:56:36,617 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:36,618 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:36,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:36,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:36,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:36,618 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:36,619 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:36,619 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:36,620 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:36,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:36,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7e37f3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:36,621 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:36,621 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,625 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:36,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:36,626 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:36,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,626 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:36,627 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:36,627 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:36,627 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:36,627 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:36,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:36,630 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:36,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:36,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:36,630 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:36,630 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:36,631 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,631 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,631 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:36,632 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:36,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,633 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:36,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:36,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:36,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:36,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:36,635 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:36,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:36,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:36,635 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:36,636 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:36,636 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:36,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:36,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:36,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:36,736 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:36,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:36,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:36,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:36,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:36,740 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:36,740 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:36,740 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:36,740 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:36,740 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,740 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:36,743 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,743 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,743 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:36,744 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:36,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:36,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:36,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:36,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:36,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:36,747 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:36,747 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:36,747 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:36,748 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:36,748 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:36,749 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 18:56:36,749 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:36,749 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:36,749 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:36,749 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:36,749 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:36,749 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:36,763 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331be574baee4714e5d8d8516fbaad1897b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331be574baee4714e5d8d8516fbaad1897b 2023-03-31 18:56:36,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:36,831 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:36,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289056855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289056855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289056856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289056959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289056959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:36,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:36,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289056959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289057162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289057162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289057163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,169 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=473, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/8bcaa33ebf564ab980af3e04fc05c6ed 2023-03-31 18:56:37,180 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/ea17a3728fd2479d8bbb5f4bf7712abc 2023-03-31 18:56:37,202 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/036c9cfc112b45aca53945f0235f7bc7 2023-03-31 18:56:37,210 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/8bcaa33ebf564ab980af3e04fc05c6ed as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8bcaa33ebf564ab980af3e04fc05c6ed 2023-03-31 18:56:37,215 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8bcaa33ebf564ab980af3e04fc05c6ed, entries=150, sequenceid=473, filesize=25.4 K 2023-03-31 18:56:37,215 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/ea17a3728fd2479d8bbb5f4bf7712abc as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ea17a3728fd2479d8bbb5f4bf7712abc 2023-03-31 18:56:37,219 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ea17a3728fd2479d8bbb5f4bf7712abc, entries=150, sequenceid=473, filesize=11.9 K 2023-03-31 18:56:37,220 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/036c9cfc112b45aca53945f0235f7bc7 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/036c9cfc112b45aca53945f0235f7bc7 2023-03-31 18:56:37,225 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/036c9cfc112b45aca53945f0235f7bc7, entries=150, sequenceid=473, filesize=11.9 K 2023-03-31 18:56:37,226 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f34b89af9bae1a93ef5b2acb275cb610 in 478ms, sequenceid=473, compaction requested=true 2023-03-31 18:56:37,226 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:37,226 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:37,227 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,227 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:37,228 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,228 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b6dc0be[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:37,229 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:37,229 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,231 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,231 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:37,231 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,231 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:37,231 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:37,231 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:37,232 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:37,232 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:37,232 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,232 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:37,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:37,235 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:37,235 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:37,235 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:37,235 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:37,235 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:37,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:37,237 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:37,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,238 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,239 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:37,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:37,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,242 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:37,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:37,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:37,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:37,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:37,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:37,242 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:37,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:37,243 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:37,243 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:37,243 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,343 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:37,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:37,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:37,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:37,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:37,347 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:37,347 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:37,347 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:37,347 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:37,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,350 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:37,350 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:37,351 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,351 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:37,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:37,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:37,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:37,354 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:37,354 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:37,354 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:37,354 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:37,354 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:37,354 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:37,354 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:37,354 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 18:56:37,355 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:37,355 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:37,355 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:37,355 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:37,355 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:37,355 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:37,367 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331163dc1b159fa45a2b5d58cb14d7ec175 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331163dc1b159fa45a2b5d58cb14d7ec175 2023-03-31 18:56:37,372 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=493, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/eba880cb1b5a4a19b4154f020db7cbc6 2023-03-31 18:56:37,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:37,468 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:37,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289057499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289057499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289057499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289057605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289057605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289057606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,780 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/1d0b1498a220480bab18c3b125bff379 2023-03-31 18:56:37,792 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/70ff5747993e4cd580938d7969a22ec6 2023-03-31 18:56:37,795 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/eba880cb1b5a4a19b4154f020db7cbc6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eba880cb1b5a4a19b4154f020db7cbc6 2023-03-31 18:56:37,798 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eba880cb1b5a4a19b4154f020db7cbc6, entries=150, sequenceid=493, filesize=25.4 K 2023-03-31 18:56:37,798 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/1d0b1498a220480bab18c3b125bff379 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1d0b1498a220480bab18c3b125bff379 2023-03-31 18:56:37,801 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1d0b1498a220480bab18c3b125bff379, entries=150, sequenceid=493, filesize=11.9 K 2023-03-31 18:56:37,802 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/70ff5747993e4cd580938d7969a22ec6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/70ff5747993e4cd580938d7969a22ec6 2023-03-31 18:56:37,805 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/70ff5747993e4cd580938d7969a22ec6, entries=150, sequenceid=493, filesize=11.9 K 2023-03-31 18:56:37,806 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for f34b89af9bae1a93ef5b2acb275cb610 in 452ms, sequenceid=493, compaction requested=true 2023-03-31 18:56:37,806 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:37,806 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:37,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:37,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:37,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:37,806 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:37,808 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,808 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:37,808 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,808 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:37,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:37,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:37,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f07f4e1[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:37,809 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:37,809 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,811 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,812 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:37,812 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:37,812 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:37,812 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,812 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:37,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:37,813 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:37,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-31 18:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:37,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:37,813 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:37,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:37,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:37,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:37,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:37,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:37,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:37,816 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:37,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:37,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:37,816 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:37,816 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:37,818 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,819 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:37,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:37,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,820 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,821 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,827 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303315dfa191a96a940c2a55d06948ee91c39 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315dfa191a96a940c2a55d06948ee91c39 2023-03-31 18:56:37,827 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:37,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:37,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:37,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:37,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,828 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:37,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:37,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:37,828 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:37,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:37,829 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,829 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,829 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289057842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289057843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289057844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,929 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:37,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:37,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:37,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:37,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:37,932 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:37,932 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:37,932 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:37,932 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:37,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,933 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,934 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:37,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:37,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,934 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:37,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:37,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:37,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:37,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:37,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:37,935 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:37,936 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:37,936 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:37,936 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:37,936 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:37,937 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,938 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:37,938 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,938 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:37,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:37,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2c6828[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:37,939 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:37,939 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,940 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,941 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:37,941 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:37,941 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:37,941 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,941 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:37,944 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:37,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:37,946 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:37,946 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:37,946 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:37,946 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:37,946 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:37,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289057947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:37,948 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:37,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:37,949 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,950 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:37,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:37,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:37,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:37,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:37,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:37,952 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:37,953 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:37,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:37,953 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:37,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:37,953 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:37,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:37,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289057950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:37,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:37,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289057950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,053 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:38,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:38,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:38,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:38,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:38,059 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:38,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:38,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:38,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,059 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,061 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,061 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,061 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,061 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:38,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:38,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:38,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:38,062 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:38,062 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:38,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:38,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:38,063 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,063 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:38,063 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:38,063 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:38,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:38,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:38,063 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:38,066 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,066 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,066 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,066 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:38,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@539213f1[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:38,068 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:38,068 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,069 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,069 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:38,069 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:38,069 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:38,069 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,069 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:38,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:38,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,071 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:38,071 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:38,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:38,073 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:38,073 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:38,073 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:38,073 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:38,073 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:38,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,075 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,076 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,078 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,081 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,081 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,081 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,081 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,081 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,081 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,081 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,081 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,081 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,081 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:38,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:38,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,082 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:38,082 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:38,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:38,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:38,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289058150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:38,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289058156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:38,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289058156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,182 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:38,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:38,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:38,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:38,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:38,185 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:38,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:38,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:38,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,185 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,187 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,187 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,187 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,188 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:38,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:38,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:38,189 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,189 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:38,189 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:38,189 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:38,190 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,191 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,191 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,191 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:38,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@369f510d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:38,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:38,192 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,193 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:38,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:38,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:38,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,194 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:38,196 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:38,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,196 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:38,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:38,199 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:38,199 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:38,199 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:38,199 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:38,199 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:38,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,200 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,202 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,203 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,205 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,205 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:38,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,206 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,206 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:38,206 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:38,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,233 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=507, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/39b1d07feee24906b29226a963579853 2023-03-31 18:56:38,306 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:38,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:38,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:38,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:38,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:38,310 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:38,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:38,310 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:38,311 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,311 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,318 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,319 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,319 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:38,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:38,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:38,321 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,321 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:38,321 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:38,321 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:38,323 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,323 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,323 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,323 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@363f1bc3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:38,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:38,325 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,326 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:38,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:38,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:38,327 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,327 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:38,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:38,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,328 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:38,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:38,331 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:38,331 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:38,331 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:38,331 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:38,331 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:38,334 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,334 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,334 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,336 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,339 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:38,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:38,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,339 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:38,340 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:38,340 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:38,440 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:38,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:38,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:38,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:38,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:38,443 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:38,443 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,443 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:38,443 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:38,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,444 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,445 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:38,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:38,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:38,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:38,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:38,447 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:38,447 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,447 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:38,448 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:38,448 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:38,449 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:38,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799d4804[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:38,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:38,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,454 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,454 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:38,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:38,455 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:38,455 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,455 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:38,456 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,456 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:38,456 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,456 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289058456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:38,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:38,459 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:38,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:38,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:38,459 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:38,459 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:38,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,461 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,462 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:38,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289058459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:38,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289058460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,466 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:38,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:38,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:38,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,467 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:38,467 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:38,466 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,567 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:38,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:38,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:38,570 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:38,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:38,570 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:38,571 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,571 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,577 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,577 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:38,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:38,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:38,578 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:38,578 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,578 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:38,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:38,580 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,580 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:38,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:38,580 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:38,580 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:38,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:38,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:38,580 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:38,582 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,582 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,582 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,582 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:38,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ad3d15d[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:38,584 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:38,584 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,585 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,585 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:38,585 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:38,585 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:38,585 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,585 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:38,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:38,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,587 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,587 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:38,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:38,589 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:38,589 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:38,589 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:38,589 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:38,589 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:38,593 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,593 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,593 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,594 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,595 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,597 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:38,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,597 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,597 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:38,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,597 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:38,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/f20d0b1993bf4e90b28ec6cd2313294e 2023-03-31 18:56:38,697 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:38,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:38,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:38,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:38,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:38,700 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:38,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,701 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:38,701 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:38,701 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,701 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,702 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,702 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,702 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,702 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:38,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:38,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:38,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:38,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:38,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:38,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:38,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:38,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:38,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:38,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:38,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:38,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:38,709 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,709 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,709 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,709 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:38,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7688f833[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:38,711 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:38,711 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,712 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:38,712 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,712 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:38,713 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,713 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:38,714 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,714 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:38,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:38,716 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:38,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:38,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:38,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:38,716 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:38,717 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,717 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,718 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,718 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,719 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:38,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:38,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,722 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:38,723 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:38,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,722 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,723 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,723 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,722 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:38,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,823 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:38,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:38,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:38,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:38,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:38,826 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:38,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:38,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:38,826 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,827 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,828 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:38,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:38,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:38,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:38,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:38,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:38,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,829 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:38,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:38,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:38,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:38,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:38,830 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:38,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:38,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:38,830 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:38,832 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,832 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,832 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,832 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:38,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:38,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38271422[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:38,834 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:38,835 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:38,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:38,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:38,836 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,836 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:38,838 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,838 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,838 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,838 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:38,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:38,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:38,841 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:38,841 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:38,842 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:38,842 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:38,842 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:38,843 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,843 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,843 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,844 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,845 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,848 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:38,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:38,848 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,848 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:38,848 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:38,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:38,948 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:38,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:38,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:38,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:38,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:38,951 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:38,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:38,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:38,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,952 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,953 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,953 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,953 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,954 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:38,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:38,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:38,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:38,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:38,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:38,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:38,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:38,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:38,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:38,955 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,955 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:38,955 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:38,955 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:38,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:38,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:38,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:38,955 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:38,957 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,957 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:38,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:38,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b69a64a[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:38,959 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:38,959 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,960 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,960 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:38,960 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:38,960 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:38,960 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,961 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:38,962 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,962 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:38,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,962 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,962 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:38,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:38,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289058960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:38,964 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:38,964 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:38,965 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:38,965 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:38,965 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:38,966 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,966 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:38,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,966 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:38,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:38,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289058964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,967 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:38,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,968 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:38,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:38,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289058966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,971 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:38,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:38,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:38,972 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:38,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:38,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:38,972 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:38,972 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:38,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:38,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:38,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:38,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:38,972 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:38,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:38,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,052 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/8cc3f722da974d218c7f69a6f4306c96 2023-03-31 18:56:39,055 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/39b1d07feee24906b29226a963579853 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/39b1d07feee24906b29226a963579853 2023-03-31 18:56:39,058 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/39b1d07feee24906b29226a963579853, entries=150, sequenceid=507, filesize=25.4 K 2023-03-31 18:56:39,059 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/f20d0b1993bf4e90b28ec6cd2313294e as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f20d0b1993bf4e90b28ec6cd2313294e 2023-03-31 18:56:39,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f20d0b1993bf4e90b28ec6cd2313294e, entries=150, sequenceid=507, filesize=11.9 K 2023-03-31 18:56:39,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/8cc3f722da974d218c7f69a6f4306c96 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8cc3f722da974d218c7f69a6f4306c96 2023-03-31 18:56:39,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8cc3f722da974d218c7f69a6f4306c96, entries=150, sequenceid=507, filesize=11.9 K 2023-03-31 18:56:39,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for f34b89af9bae1a93ef5b2acb275cb610 in 1254ms, sequenceid=507, compaction requested=true 2023-03-31 18:56:39,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:39,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:39,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:39,067 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-31 18:56:39,067 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-31 18:56:39,070 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86086 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-31 18:56:39,070 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 183072 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-31 18:56:39,070 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/B is initiating minor compaction (all files) 2023-03-31 18:56:39,071 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/A is initiating minor compaction (all files) 2023-03-31 18:56:39,071 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/B in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,071 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/A in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,071 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/8d25f0008b834586ba94717d621debda, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/67ff4b921b03467abaa9a6a0f64c6947, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/af4ae60919e24b8d9f88d77f1412e18e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ab513c6159ae495e9c1b41577f351ca9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ea17a3728fd2479d8bbb5f4bf7712abc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1d0b1498a220480bab18c3b125bff379, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f20d0b1993bf4e90b28ec6cd2313294e] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=84.1 K 2023-03-31 18:56:39,071 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/e82ee6458a7847b7861e7b4199556689, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/063ce5242abc4d88b8f84d7f42ad1c8e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/98ccbe36cfe14e4a8159def08a1319aa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/fd6bb06aab9e4a468bd709a0eb78ae78, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8bcaa33ebf564ab980af3e04fc05c6ed, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eba880cb1b5a4a19b4154f020db7cbc6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/39b1d07feee24906b29226a963579853] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=178.8 K 2023-03-31 18:56:39,071 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 8d25f0008b834586ba94717d621debda, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680288993622 2023-03-31 18:56:39,071 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting e82ee6458a7847b7861e7b4199556689, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680288993622 2023-03-31 18:56:39,072 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 67ff4b921b03467abaa9a6a0f64c6947, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680288993743 2023-03-31 18:56:39,072 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 063ce5242abc4d88b8f84d7f42ad1c8e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680288993743 2023-03-31 18:56:39,072 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting af4ae60919e24b8d9f88d77f1412e18e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680288994751 2023-03-31 18:56:39,072 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:39,072 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 98ccbe36cfe14e4a8159def08a1319aa, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680288994751 2023-03-31 18:56:39,073 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ab513c6159ae495e9c1b41577f351ca9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680288995898 2023-03-31 18:56:39,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:39,073 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting fd6bb06aab9e4a468bd709a0eb78ae78, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680288995898 2023-03-31 18:56:39,073 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting ea17a3728fd2479d8bbb5f4bf7712abc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680288996218 2023-03-31 18:56:39,073 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8bcaa33ebf564ab980af3e04fc05c6ed, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680288996218 2023-03-31 18:56:39,074 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 1d0b1498a220480bab18c3b125bff379, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680288996851 2023-03-31 18:56:39,074 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting eba880cb1b5a4a19b4154f020db7cbc6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680288996851 2023-03-31 18:56:39,074 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting f20d0b1993bf4e90b28ec6cd2313294e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680288997475 2023-03-31 18:56:39,074 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 39b1d07feee24906b29226a963579853, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680288997475 2023-03-31 18:56:39,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:39,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:39,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:39,076 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:39,076 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,077 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:39,077 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:39,077 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,077 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,078 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,078 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,079 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,079 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:39,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:39,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:39,080 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:39,080 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,080 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:39,081 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:39,081 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:39,081 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,081 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:39,081 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-31 18:56:39,081 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:39,081 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:39,081 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:39,081 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:39,081 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:39,082 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:39,095 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#B#compaction#615 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-03-31 18:56:39,103 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#A#compaction#616 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-31 18:56:39,130 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303318321d657340f4dfaa31401a8dfb56805 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318321d657340f4dfaa31401a8dfb56805 2023-03-31 18:56:39,140 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=525, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/24c0d09569d74661bb97f8c2a6a027ce 2023-03-31 18:56:39,148 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/a369daf5a3df4d5f895ca7633545d767 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/a369daf5a3df4d5f895ca7633545d767 2023-03-31 18:56:39,152 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/bb241f57fd764c68ab80d27f054500f6 2023-03-31 18:56:39,154 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/A of f34b89af9bae1a93ef5b2acb275cb610 into a369daf5a3df4d5f895ca7633545d767(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:39,154 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,154 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/A, priority=9, startTime=1680288999067; duration=0sec 2023-03-31 18:56:39,154 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:39,154 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-31 18:56:39,156 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86052 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-31 18:56:39,156 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/C is initiating minor compaction (all files) 2023-03-31 18:56:39,156 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/C in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,156 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/34cfcff182144f649d140f0ea66df39c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/701665bcef6b415f86604569d34eddc5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/735487691bbc4fb1a414b9d9b14a8e5c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/13ce14112d8543ed8536fe39bba80363, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/036c9cfc112b45aca53945f0235f7bc7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/70ff5747993e4cd580938d7969a22ec6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8cc3f722da974d218c7f69a6f4306c96] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=84.0 K 2023-03-31 18:56:39,157 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 34cfcff182144f649d140f0ea66df39c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680288993622 2023-03-31 18:56:39,157 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 701665bcef6b415f86604569d34eddc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680288993743 2023-03-31 18:56:39,157 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 735487691bbc4fb1a414b9d9b14a8e5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680288994751 2023-03-31 18:56:39,158 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 13ce14112d8543ed8536fe39bba80363, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680288995898 2023-03-31 18:56:39,158 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 036c9cfc112b45aca53945f0235f7bc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680288996218 2023-03-31 18:56:39,158 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 70ff5747993e4cd580938d7969a22ec6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680288996851 2023-03-31 18:56:39,159 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 8cc3f722da974d218c7f69a6f4306c96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680288997475 2023-03-31 18:56:39,161 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/e3fc9a7ae0fa497d8d96049942e15fac 2023-03-31 18:56:39,166 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/24c0d09569d74661bb97f8c2a6a027ce as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/24c0d09569d74661bb97f8c2a6a027ce 2023-03-31 18:56:39,182 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/24c0d09569d74661bb97f8c2a6a027ce, entries=150, sequenceid=525, filesize=25.4 K 2023-03-31 18:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/bb241f57fd764c68ab80d27f054500f6 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/bb241f57fd764c68ab80d27f054500f6 2023-03-31 18:56:39,183 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#C#compaction#620 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-31 18:56:39,186 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/bb241f57fd764c68ab80d27f054500f6, entries=150, sequenceid=525, filesize=11.9 K 2023-03-31 18:56:39,186 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/e3fc9a7ae0fa497d8d96049942e15fac as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/e3fc9a7ae0fa497d8d96049942e15fac 2023-03-31 18:56:39,191 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/e3fc9a7ae0fa497d8d96049942e15fac, entries=150, sequenceid=525, filesize=11.9 K 2023-03-31 18:56:39,192 INFO [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for f34b89af9bae1a93ef5b2acb275cb610 in 111ms, sequenceid=525, compaction requested=false 2023-03-31 18:56:39,192 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,192 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:39,193 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/2428150d4f724ecf91a9d636eb2b6a03 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2428150d4f724ecf91a9d636eb2b6a03 2023-03-31 18:56:39,193 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:39,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f94d343[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:39,196 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:39,196 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,197 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,197 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:39,197 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:39,197 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:39,197 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/C of f34b89af9bae1a93ef5b2acb275cb610 into 2428150d4f724ecf91a9d636eb2b6a03(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:39,197 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,198 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:39,198 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,198 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/C, priority=9, startTime=1680288999067; duration=0sec 2023-03-31 18:56:39,198 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:39,202 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:39,202 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,202 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,202 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:39,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:39,205 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:39,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:39,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:39,205 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:39,205 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:39,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,207 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,209 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,211 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:39,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:39,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:39,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,211 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:39,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,212 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:39,211 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,312 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:39,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:39,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:39,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:39,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:39,315 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:39,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:39,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:39,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,316 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,317 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,317 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,318 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:39,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:39,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:39,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:39,319 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:39,319 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,320 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:39,320 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,320 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:39,322 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,322 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,322 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,322 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:39,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eecfbe2[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:39,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:39,324 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,325 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,325 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:39,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:39,326 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,326 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:39,326 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:39,327 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,327 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:39,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:39,331 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:39,331 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:39,331 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:39,331 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:39,331 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:39,332 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,332 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,332 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,333 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,334 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,337 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,337 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,337 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,337 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:39,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:39,337 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,337 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:39,338 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:39,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,338 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:39,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,338 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,438 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:39,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:39,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:39,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:39,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:39,445 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:39,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:39,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:39,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,445 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,446 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,446 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,447 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,447 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:39,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:39,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:39,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:39,448 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:39,448 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:39,449 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,449 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:39,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:39,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:39,449 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:39,450 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,450 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,451 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,451 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:39,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:39,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6a7e19[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:39,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:39,452 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,453 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,454 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:39,454 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:39,454 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:39,454 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,454 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:39,457 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,457 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:39,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:39,460 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:39,460 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:39,460 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:39,460 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:39,460 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:39,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,462 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,463 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,464 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,467 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,467 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,467 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:39,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:39,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,467 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,467 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:39,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:39,467 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,467 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,468 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,468 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,468 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,468 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,467 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:39,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,526 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/81607bd668854e7d940b17fb61746676 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/81607bd668854e7d940b17fb61746676 2023-03-31 18:56:39,529 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/B of f34b89af9bae1a93ef5b2acb275cb610 into 81607bd668854e7d940b17fb61746676(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:39,530 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,530 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/B, priority=9, startTime=1680288999067; duration=0sec 2023-03-31 18:56:39,530 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:39,568 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:39,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:39,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:39,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:39,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:39,571 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:39,571 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,572 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:39,572 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:39,572 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,572 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,573 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,573 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,574 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,574 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:39,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:39,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:39,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:39,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:39,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:39,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:39,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:39,575 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:39,575 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,575 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:39,576 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,576 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:39,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:39,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:39,576 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:39,578 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,578 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,579 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:39,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d7d468[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:39,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:39,580 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,581 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:39,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:39,581 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,581 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:39,581 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:39,583 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,583 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:39,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:39,586 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:39,586 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:39,586 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:39,586 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:39,586 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:39,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,590 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,591 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,591 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,592 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,595 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:39,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:39,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:39,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,595 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,596 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:39,596 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:39,596 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,696 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:39,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:39,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:39,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:39,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:39,699 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:39,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:39,699 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:39,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,700 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,701 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,701 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,702 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,702 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:39,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:39,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:39,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:39,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:39,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:39,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:39,703 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,703 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:39,703 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:39,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,704 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:39,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:39,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:39,704 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:39,705 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,705 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,706 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,706 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:39,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71700afa[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:39,707 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:39,708 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,710 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:39,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:39,710 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:39,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,710 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:39,711 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:39,711 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,711 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,711 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:39,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:39,714 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:39,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:39,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:39,714 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:39,714 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:39,716 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,716 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,716 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,716 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,718 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:39,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:39,724 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:39,724 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,725 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:39,725 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:39,725 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,725 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,825 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:39,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:39,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:39,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:39,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:39,828 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:39,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:39,828 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:39,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,829 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,832 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,832 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,832 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,832 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:39,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:39,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:39,833 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:39,833 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:39,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:39,834 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,834 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:39,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:39,834 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,834 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:39,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:39,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:39,834 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:39,836 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,836 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:39,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@281adcdd[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:39,837 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:39,838 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,839 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,839 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:39,839 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:39,839 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,839 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:39,839 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:39,841 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,841 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:39,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,841 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,841 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:39,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:39,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:39,843 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:39,843 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:39,843 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:39,844 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:39,844 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:39,846 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,846 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,846 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,847 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,848 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,851 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:39,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:39,851 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:39,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,851 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:39,851 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:39,952 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:39,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:39,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:39,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:39,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:39,955 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:39,955 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,955 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:39,955 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:39,955 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,956 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,957 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,957 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,957 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,957 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:39,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:39,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:39,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:39,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:39,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:39,958 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:39,959 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,959 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:39,959 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:39,959 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:39,959 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:39,959 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:39,959 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:39,959 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,959 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:39,959 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:39,960 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,960 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,961 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,961 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:39,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45edf571[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:39,963 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:39,963 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,965 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,965 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:39,965 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:39,965 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:39,965 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,965 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:39,966 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:39,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,966 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:39,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:39,969 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:39,969 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:39,969 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:39,969 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:39,969 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:39,971 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,971 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:39,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,971 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,972 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,973 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:39,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:39,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:39,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:39,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:39,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:39,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:39,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:39,979 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:39,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:39,979 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:39,979 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:39,979 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:39,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:39,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:39,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:39,996 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 18:56:39,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:39,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:39,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:39,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:39,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:39,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:40,009 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331f52f1899f97d458aa5680006574c0c5a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f52f1899f97d458aa5680006574c0c5a 2023-03-31 18:56:40,014 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=544, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/15bace5b09a049e1a80ec685131f2777 2023-03-31 18:56:40,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/a2eb87dade0843789775bf11338ab842 2023-03-31 18:56:40,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680289000002, max timestamp=1680289000028Num uniques 859; 2023-03-31 18:56:40,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:40,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680289000002, max timestamp=1680289000028Num uniques 859; 2023-03-31 18:56:40,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:40,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680289000002, max timestamp=1680289000028Num uniques 859; 2023-03-31 18:56:40,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 18:56:40,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:40,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289060053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:40,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:40,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289060055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289060055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,079 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:40,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:40,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:40,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:40,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:40,083 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:40,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:40,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:40,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:40,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,083 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,084 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:40,084 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:40,085 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,085 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:40,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:40,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:40,086 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:40,086 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:40,086 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:40,086 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:40,088 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:40,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:40,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23ff1d30[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:40,090 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:40,090 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,091 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:40,091 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,091 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:40,094 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:40,094 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:40,094 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:40,094 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:40,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:40,096 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:40,096 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:40,096 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:40,096 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:40,096 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:40,098 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,098 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,098 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:40,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:40,098 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,099 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:40,102 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:40,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:40,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:40,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:40,102 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,103 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:40,103 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:40,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:40,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34951] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289060156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:40,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289060160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:40,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289060160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,203 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:40,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:40,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:40,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:40,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:40,206 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:40,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:40,206 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:40,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:40,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,207 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,208 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,208 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:40,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:40,209 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,209 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:40,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 18:56:40,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:40,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:40,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:40,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:40,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:40,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:40,210 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:40,210 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:40,210 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:40,210 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:40,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:40,210 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:40,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:40,211 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:40,213 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,213 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:40,213 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,213 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:40,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:40,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:40,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:40,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:40,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5229e46f[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:40,215 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:40,216 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,217 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:40,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:40,217 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:40,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,218 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:40,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:40,219 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:40,219 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:40,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:40,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:40,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:40,223 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:40,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:40,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:40,223 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:40,223 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,224 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:40,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:40,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:40,225 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:40,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,227 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,230 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,230 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,230 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,230 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:40,230 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:40,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:40,230 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:40,230 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:40,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:40,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:40,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:40,231 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,231 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:40,231 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,231 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:40,231 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:40,331 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:40,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:40,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 18:56:40,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 18:56:40,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 18:56:40,334 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 18:56:40,334 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:40,334 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 18:56:40,334 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 18:56:40,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,335 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,336 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,336 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:40,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:40,336 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,336 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 18:56:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 18:56:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 18:56:40,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 18:56:40,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 18:56:40,337 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 18:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 18:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 18:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 18:56:40,338 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:40,338 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. started... 2023-03-31 18:56:40,338 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. as already flushing 2023-03-31 18:56:40,338 DEBUG [rs(jenkins-hbase4.apache.org,34951,1680288816339)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 18:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 18:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 18:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 18:56:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,340 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 18:56:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:40,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34951,1680288816339' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 18:56:40,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2b36c3[Count = 0] remaining members to acquire global barrier 2023-03-31 18:56:40,342 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 18:56:40,342 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,343 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,343 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 18:56:40,343 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 18:56:40,343 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34951,1680288816339' in zk 2023-03-31 18:56:40,343 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,343 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 18:56:40,346 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 18:56:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,34951,1680288816339' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 18:56:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34951,1680288816339': 2023-03-31 18:56:40,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34951,1680288816339' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 18:56:40,348 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 18:56:40,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 18:56:40,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 18:56:40,348 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 18:56:40,348 INFO [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 18:56:40,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,349 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 18:56:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 18:56:40,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 18:56:40,350 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 18:56:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,351 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 18:56:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 18:56:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,353 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 18:56:40,353 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 18:56:40,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 18:56:40,354 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,354 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 18:56:40,354 DEBUG [(jenkins-hbase4.apache.org,35737,1680288815092)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 18:56:40,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 18:56:40,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 18:56:40,354 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 18:56:40,354 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,354 DEBUG [Thread-3981] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 18:56:40,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 18:56:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 18:56:40,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 18:56:40,354 DEBUG [Thread-3981] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 18:56:40,354 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,354 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 18:56:40,354 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 18:56:40,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:40,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33008 deadline: 1680289060361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:40,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33020 deadline: 1680289060363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 18:56:40,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34951] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33016 deadline: 1680289060363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:40,368 DEBUG [Thread-3982] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:63858 2023-03-31 18:56:40,368 DEBUG [Thread-3982] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:40,368 DEBUG [Thread-3986] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e6a6f3c to 127.0.0.1:63858 2023-03-31 18:56:40,368 DEBUG [Thread-3986] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:40,371 DEBUG [Thread-3984] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:63858 2023-03-31 18:56:40,371 DEBUG [Thread-3984] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:40,371 DEBUG [Thread-3988] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a470a3d to 127.0.0.1:63858 2023-03-31 18:56:40,371 DEBUG [Thread-3988] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:40,372 DEBUG [Thread-3990] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bb72437 to 127.0.0.1:63858 2023-03-31 18:56:40,372 DEBUG [Thread-3990] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:40,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/4a61a23553c1403fb9af01743778a469 2023-03-31 18:56:40,450 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/15bace5b09a049e1a80ec685131f2777 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/15bace5b09a049e1a80ec685131f2777 2023-03-31 18:56:40,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/15bace5b09a049e1a80ec685131f2777, entries=350, sequenceid=544, filesize=52.9 K 2023-03-31 18:56:40,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/a2eb87dade0843789775bf11338ab842 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a2eb87dade0843789775bf11338ab842 2023-03-31 18:56:40,454 DEBUG [Thread-3981] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 18:56:40,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 18:56:40,456 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a2eb87dade0843789775bf11338ab842, entries=250, sequenceid=544, filesize=16.7 K 2023-03-31 18:56:40,457 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/4a61a23553c1403fb9af01743778a469 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/4a61a23553c1403fb9af01743778a469 2023-03-31 18:56:40,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/4a61a23553c1403fb9af01743778a469, entries=150, sequenceid=544, filesize=11.9 K 2023-03-31 18:56:40,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f34b89af9bae1a93ef5b2acb275cb610 in 464ms, sequenceid=544, compaction requested=true 2023-03-31 18:56:40,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:40,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:40,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:40,460 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:40,460 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:40,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:40,461 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107294 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:40,461 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42460 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:40,461 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/A is initiating minor compaction (all files) 2023-03-31 18:56:40,461 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/B is initiating minor compaction (all files) 2023-03-31 18:56:40,461 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/A in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:40,461 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/B in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:40,461 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/a369daf5a3df4d5f895ca7633545d767, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/24c0d09569d74661bb97f8c2a6a027ce, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/15bace5b09a049e1a80ec685131f2777] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=104.8 K 2023-03-31 18:56:40,461 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/81607bd668854e7d940b17fb61746676, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/bb241f57fd764c68ab80d27f054500f6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a2eb87dade0843789775bf11338ab842] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=41.5 K 2023-03-31 18:56:40,461 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting a369daf5a3df4d5f895ca7633545d767, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680288997475 2023-03-31 18:56:40,461 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 81607bd668854e7d940b17fb61746676, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680288997475 2023-03-31 18:56:40,461 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 24c0d09569d74661bb97f8c2a6a027ce, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680288997839 2023-03-31 18:56:40,461 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting bb241f57fd764c68ab80d27f054500f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680288997839 2023-03-31 18:56:40,462 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] compactions.Compactor(207): Compacting 15bace5b09a049e1a80ec685131f2777, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680288999985 2023-03-31 18:56:40,462 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting a2eb87dade0843789775bf11338ab842, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680288999990 2023-03-31 18:56:40,467 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#B#compaction#625 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-03-31 18:56:40,469 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#A#compaction#624 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-31 18:56:40,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34951] regionserver.HRegion(9158): Flush requested on f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:40,666 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-31 18:56:40,666 DEBUG [Thread-3971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03ced4ba to 127.0.0.1:63858 2023-03-31 18:56:40,666 DEBUG [Thread-3971] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:40,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:40,667 DEBUG [Thread-3975] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59185ba7 to 127.0.0.1:63858 2023-03-31 18:56:40,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 18:56:40,667 DEBUG [Thread-3975] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:40,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:40,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680289000042, max timestamp=1680289000664Num uniques 1018; 2023-03-31 18:56:40,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680289000042, max timestamp=1680289000664Num uniques 1018; 2023-03-31 18:56:40,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 18:56:40,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:40,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 18:56:40,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:40,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 18:56:40,669 DEBUG [Thread-3977] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062b3caa to 127.0.0.1:63858 2023-03-31 18:56:40,669 DEBUG [Thread-3977] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:40,679 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303315fe4b74cc0c84104b0064ca8b501f9ff to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315fe4b74cc0c84104b0064ca8b501f9ff 2023-03-31 18:56:40,781 DEBUG [Thread-3973] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63da10ca to 127.0.0.1:63858 2023-03-31 18:56:40,781 DEBUG [Thread-3973] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:40,877 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/030a6916dd414fcbb2ffe59bb2c2c300 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/030a6916dd414fcbb2ffe59bb2c2c300 2023-03-31 18:56:40,880 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/B of f34b89af9bae1a93ef5b2acb275cb610 into 030a6916dd414fcbb2ffe59bb2c2c300(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:40,880 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:40,880 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/B, priority=13, startTime=1680289000460; duration=0sec 2023-03-31 18:56:40,880 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 18:56:40,880 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 18:56:40,881 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 18:56:40,881 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1912): f34b89af9bae1a93ef5b2acb275cb610/C is initiating minor compaction (all files) 2023-03-31 18:56:40,881 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f34b89af9bae1a93ef5b2acb275cb610/C in TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:40,881 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2428150d4f724ecf91a9d636eb2b6a03, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/e3fc9a7ae0fa497d8d96049942e15fac, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/4a61a23553c1403fb9af01743778a469] into tmpdir=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp, totalSize=36.7 K 2023-03-31 18:56:40,882 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 2428150d4f724ecf91a9d636eb2b6a03, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680288997475 2023-03-31 18:56:40,882 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting e3fc9a7ae0fa497d8d96049942e15fac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680288997839 2023-03-31 18:56:40,882 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] compactions.Compactor(207): Compacting 4a61a23553c1403fb9af01743778a469, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680288999990 2023-03-31 18:56:40,886 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] throttle.PressureAwareThroughputController(145): f34b89af9bae1a93ef5b2acb275cb610#C#compaction#627 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-31 18:56:41,083 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=583, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/88a023b7ebff49269bbf8a5ca422ced8 2023-03-31 18:56:41,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/7833a630fb034cdbb72dcac48bd00409 2023-03-31 18:56:41,293 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/0c8a07ed7d4f43fa9d394ac51c708b65 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/0c8a07ed7d4f43fa9d394ac51c708b65 2023-03-31 18:56:41,296 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/C of f34b89af9bae1a93ef5b2acb275cb610 into 0c8a07ed7d4f43fa9d394ac51c708b65(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 18:56:41,296 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:41,297 INFO [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/C, priority=13, startTime=1680289000460; duration=0sec 2023-03-31 18:56:41,297 DEBUG [RS:0;jenkins-hbase4:34951-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:41,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/38ffd8a07ab34d8ea74340f61e99a25a 2023-03-31 18:56:41,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/88a023b7ebff49269bbf8a5ca422ced8 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/88a023b7ebff49269bbf8a5ca422ced8 2023-03-31 18:56:41,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/88a023b7ebff49269bbf8a5ca422ced8, entries=150, sequenceid=583, filesize=25.4 K 2023-03-31 18:56:41,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/7833a630fb034cdbb72dcac48bd00409 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/7833a630fb034cdbb72dcac48bd00409 2023-03-31 18:56:41,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/7833a630fb034cdbb72dcac48bd00409, entries=150, sequenceid=583, filesize=11.9 K 2023-03-31 18:56:41,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/38ffd8a07ab34d8ea74340f61e99a25a as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/38ffd8a07ab34d8ea74340f61e99a25a 2023-03-31 18:56:41,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/38ffd8a07ab34d8ea74340f61e99a25a, entries=150, sequenceid=583, filesize=11.9 K 2023-03-31 18:56:41,513 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~198.67 KB/203440, currentSize=13.42 KB/13740 for f34b89af9bae1a93ef5b2acb275cb610 in 848ms, sequenceid=583, compaction requested=false 2023-03-31 18:56:41,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:41,687 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/885e7f69fde74339a1aff62069d384b4 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/885e7f69fde74339a1aff62069d384b4 2023-03-31 18:56:41,690 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f34b89af9bae1a93ef5b2acb275cb610/A of f34b89af9bae1a93ef5b2acb275cb610 into 885e7f69fde74339a1aff62069d384b4(size=26.5 K), total size for store is 51.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-31 18:56:41,690 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:41,690 INFO [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610., storeName=f34b89af9bae1a93ef5b2acb275cb610/A, priority=13, startTime=1680289000460; duration=1sec 2023-03-31 18:56:41,690 DEBUG [RS:0;jenkins-hbase4:34951-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 18:56:45,190 DEBUG [Thread-3979] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66b5b1f3 to 127.0.0.1:63858 2023-03-31 18:56:45,190 DEBUG [Thread-3979] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:45,190 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 18:56:45,190 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 118 2023-03-31 18:56:45,190 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-03-31 18:56:45,190 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 117 2023-03-31 18:56:45,190 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 103 2023-03-31 18:56:45,190 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(394): wrote 117 2023-03-31 18:56:45,190 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 18:56:45,190 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 4136 2023-03-31 18:56:45,190 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 4036 2023-03-31 18:56:45,191 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 4045 2023-03-31 18:56:45,191 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 4142 2023-03-31 18:56:45,191 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(398): read 4021 2023-03-31 18:56:45,191 INFO [Listener at localhost/43327] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 18:56:45,191 INFO [Listener at localhost/43327] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 18:56:45,191 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3bafb41f to 127.0.0.1:63858 2023-03-31 18:56:45,191 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:45,191 INFO [Listener at localhost/43327] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 18:56:45,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 18:56:45,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:45,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 18:56:45,195 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680289005195"}]},"ts":"1680289005195"} 2023-03-31 18:56:45,195 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 18:56:45,197 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 18:56:45,197 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, UNASSIGN}] 2023-03-31 18:56:45,199 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, UNASSIGN 2023-03-31 18:56:45,199 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=f34b89af9bae1a93ef5b2acb275cb610, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:45,199 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680289005199"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680289005199"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680289005199"}]},"ts":"1680289005199"} 2023-03-31 18:56:45,200 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339}] 2023-03-31 18:56:45,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 18:56:45,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:45,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f34b89af9bae1a93ef5b2acb275cb610, disabling compactions & flushes 2023-03-31 18:56:45,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:45,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:45,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. after waiting 0 ms 2023-03-31 18:56:45,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:45,352 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing f34b89af9bae1a93ef5b2acb275cb610 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-31 18:56:45,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=A 2023-03-31 18:56:45,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:45,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=B 2023-03-31 18:56:45,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:45,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f34b89af9bae1a93ef5b2acb275cb610, store=C 2023-03-31 18:56:45,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 18:56:45,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 18:56:45,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331aca054d543a743b6a1d98c2133396b06 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331aca054d543a743b6a1d98c2133396b06 2023-03-31 18:56:45,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 18:56:46,166 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=592, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/c7bc148d72ff4a83856bcc59fca04291 2023-03-31 18:56:46,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 18:56:46,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/4cf88803a4784c788e86f3ca45566fea 2023-03-31 18:56:46,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/681c538dc9fe4a9598a7111d79801575 2023-03-31 18:56:46,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/A/c7bc148d72ff4a83856bcc59fca04291 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c7bc148d72ff4a83856bcc59fca04291 2023-03-31 18:56:46,586 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c7bc148d72ff4a83856bcc59fca04291, entries=150, sequenceid=592, filesize=25.4 K 2023-03-31 18:56:46,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/B/4cf88803a4784c788e86f3ca45566fea as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/4cf88803a4784c788e86f3ca45566fea 2023-03-31 18:56:46,589 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/4cf88803a4784c788e86f3ca45566fea, entries=150, sequenceid=592, filesize=11.9 K 2023-03-31 18:56:46,590 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/.tmp/C/681c538dc9fe4a9598a7111d79801575 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/681c538dc9fe4a9598a7111d79801575 2023-03-31 18:56:46,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/681c538dc9fe4a9598a7111d79801575, entries=150, sequenceid=592, filesize=11.9 K 2023-03-31 18:56:46,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for f34b89af9bae1a93ef5b2acb275cb610 in 1241ms, sequenceid=592, compaction requested=true 2023-03-31 18:56:46,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/44444be64c764613a058cab332105e1f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df4d8569e3224c71b0924fb4ede7bb6a, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8a056cb0bbf84b919ec78a70fcd3e823, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c1ff0a7c8a90422d9cd8a38a5b064a84, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4f662ded670b45859488f567fa1892dd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/69cbc3672fc14d56a7c5ff1ba74cb104, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/63bab03e5eaf49dd970805ee7c6570a3, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5603991593534077ab3db86b006193bb, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8f6c630ec1de463799da3289f694ba90, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/d25b046269544aaab3604ef08d93cdb8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5a43151c0c0c4b7f86fda16adc83e87f, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/0e0924523bc54205b1a1707bed5d05f8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/b115b79ab175439687681515cceacf31, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df000f06b43a44b8bb3ffa75acb7f264, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4d8350e9d6c84853adf4e770dfcef492, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c323571baf534a54b4530cfcb180c1c5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eb752248766440f2801eb41badc5972d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/65975a42fb2a4d90a179aa70283699e1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/3e77f73cbcbb4cb09eca4c0a05892a32, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c23b5c4d0283411580327ec6289e09cb, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/7d37f70ee0ff4873a213a04ee04f4eee, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/e82ee6458a7847b7861e7b4199556689, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/91016faf983d44da81803cf1eb3ca2ff, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/063ce5242abc4d88b8f84d7f42ad1c8e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/98ccbe36cfe14e4a8159def08a1319aa, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/fd6bb06aab9e4a468bd709a0eb78ae78, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8bcaa33ebf564ab980af3e04fc05c6ed, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eba880cb1b5a4a19b4154f020db7cbc6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/a369daf5a3df4d5f895ca7633545d767, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/39b1d07feee24906b29226a963579853, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/24c0d09569d74661bb97f8c2a6a027ce, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/15bace5b09a049e1a80ec685131f2777] to archive 2023-03-31 18:56:46,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:56:46,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/44444be64c764613a058cab332105e1f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/44444be64c764613a058cab332105e1f 2023-03-31 18:56:46,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df4d8569e3224c71b0924fb4ede7bb6a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df4d8569e3224c71b0924fb4ede7bb6a 2023-03-31 18:56:46,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8a056cb0bbf84b919ec78a70fcd3e823 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8a056cb0bbf84b919ec78a70fcd3e823 2023-03-31 18:56:46,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c1ff0a7c8a90422d9cd8a38a5b064a84 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c1ff0a7c8a90422d9cd8a38a5b064a84 2023-03-31 18:56:46,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4f662ded670b45859488f567fa1892dd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4f662ded670b45859488f567fa1892dd 2023-03-31 18:56:46,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/69cbc3672fc14d56a7c5ff1ba74cb104 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/69cbc3672fc14d56a7c5ff1ba74cb104 2023-03-31 18:56:46,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/63bab03e5eaf49dd970805ee7c6570a3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/63bab03e5eaf49dd970805ee7c6570a3 2023-03-31 18:56:46,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5603991593534077ab3db86b006193bb to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5603991593534077ab3db86b006193bb 2023-03-31 18:56:46,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8f6c630ec1de463799da3289f694ba90 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8f6c630ec1de463799da3289f694ba90 2023-03-31 18:56:46,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/d25b046269544aaab3604ef08d93cdb8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/d25b046269544aaab3604ef08d93cdb8 2023-03-31 18:56:46,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5a43151c0c0c4b7f86fda16adc83e87f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/5a43151c0c0c4b7f86fda16adc83e87f 2023-03-31 18:56:46,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/0e0924523bc54205b1a1707bed5d05f8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/0e0924523bc54205b1a1707bed5d05f8 2023-03-31 18:56:46,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/b115b79ab175439687681515cceacf31 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/b115b79ab175439687681515cceacf31 2023-03-31 18:56:46,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df000f06b43a44b8bb3ffa75acb7f264 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/df000f06b43a44b8bb3ffa75acb7f264 2023-03-31 18:56:46,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4d8350e9d6c84853adf4e770dfcef492 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/4d8350e9d6c84853adf4e770dfcef492 2023-03-31 18:56:46,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c323571baf534a54b4530cfcb180c1c5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c323571baf534a54b4530cfcb180c1c5 2023-03-31 18:56:46,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eb752248766440f2801eb41badc5972d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eb752248766440f2801eb41badc5972d 2023-03-31 18:56:46,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/65975a42fb2a4d90a179aa70283699e1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/65975a42fb2a4d90a179aa70283699e1 2023-03-31 18:56:46,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/3e77f73cbcbb4cb09eca4c0a05892a32 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/3e77f73cbcbb4cb09eca4c0a05892a32 2023-03-31 18:56:46,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c23b5c4d0283411580327ec6289e09cb to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c23b5c4d0283411580327ec6289e09cb 2023-03-31 18:56:46,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/7d37f70ee0ff4873a213a04ee04f4eee to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/7d37f70ee0ff4873a213a04ee04f4eee 2023-03-31 18:56:46,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/e82ee6458a7847b7861e7b4199556689 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/e82ee6458a7847b7861e7b4199556689 2023-03-31 18:56:46,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/91016faf983d44da81803cf1eb3ca2ff to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/91016faf983d44da81803cf1eb3ca2ff 2023-03-31 18:56:46,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/063ce5242abc4d88b8f84d7f42ad1c8e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/063ce5242abc4d88b8f84d7f42ad1c8e 2023-03-31 18:56:46,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/98ccbe36cfe14e4a8159def08a1319aa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/98ccbe36cfe14e4a8159def08a1319aa 2023-03-31 18:56:46,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/fd6bb06aab9e4a468bd709a0eb78ae78 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/fd6bb06aab9e4a468bd709a0eb78ae78 2023-03-31 18:56:46,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8bcaa33ebf564ab980af3e04fc05c6ed to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/8bcaa33ebf564ab980af3e04fc05c6ed 2023-03-31 18:56:46,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eba880cb1b5a4a19b4154f020db7cbc6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/eba880cb1b5a4a19b4154f020db7cbc6 2023-03-31 18:56:46,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/a369daf5a3df4d5f895ca7633545d767 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/a369daf5a3df4d5f895ca7633545d767 2023-03-31 18:56:46,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/39b1d07feee24906b29226a963579853 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/39b1d07feee24906b29226a963579853 2023-03-31 18:56:46,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/24c0d09569d74661bb97f8c2a6a027ce to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/24c0d09569d74661bb97f8c2a6a027ce 2023-03-31 18:56:46,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/15bace5b09a049e1a80ec685131f2777 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/15bace5b09a049e1a80ec685131f2777 2023-03-31 18:56:46,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/440737de07014187a68b85a991981ee4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/4f5f893647a3495f98da3f48697bee36, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1878856c38e14c2e9a444b470bcb1637, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f4f96058fd6e445a858c63a2da5abc24, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e36fd4df41c6493aa474b2ee1ff44509, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ee4093a270c648cba1fff0cf834d6511, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/0b4d88e895b843d098f475c93632c491, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/6fb70033454e4d77ad253b7b6d79d9cb, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73f6b473ebd34e89b66b065a69ced439, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/eb56197c414f499b93c496092c7067c8, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9ff411089d434bd68a8677d942d32738, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e05bf0a0ea914454b7e3a714639d88e1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2e34044873724ad79d7fcae000cf643d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73c5ef4e8be740f9bf929200d02581d4, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a6a06d81c12a49c4aa3088b238a41efd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9456146bac7e4d3ebc350ad617442780, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/43aee40f4cf34d6388ae2d0d7c229159, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/d492fd3079064ef6959d791b33c02233, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2a32d880f2fa4a22a41208cfafe3e78e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/92caf5deb6444d839c3d4df34318a318, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/607dbd5229e748908c3ab73bcd1218a9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/c37e93b1d4d24444a52d42774e5f8672, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/8d25f0008b834586ba94717d621debda, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9e14fc7cfd3d42229f9d6208abd22c8c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/67ff4b921b03467abaa9a6a0f64c6947, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/af4ae60919e24b8d9f88d77f1412e18e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ab513c6159ae495e9c1b41577f351ca9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ea17a3728fd2479d8bbb5f4bf7712abc, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1d0b1498a220480bab18c3b125bff379, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/81607bd668854e7d940b17fb61746676, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f20d0b1993bf4e90b28ec6cd2313294e, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/bb241f57fd764c68ab80d27f054500f6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a2eb87dade0843789775bf11338ab842] to archive 2023-03-31 18:56:46,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:56:46,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/440737de07014187a68b85a991981ee4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/440737de07014187a68b85a991981ee4 2023-03-31 18:56:46,629 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/4f5f893647a3495f98da3f48697bee36 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/4f5f893647a3495f98da3f48697bee36 2023-03-31 18:56:46,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1878856c38e14c2e9a444b470bcb1637 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1878856c38e14c2e9a444b470bcb1637 2023-03-31 18:56:46,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f4f96058fd6e445a858c63a2da5abc24 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f4f96058fd6e445a858c63a2da5abc24 2023-03-31 18:56:46,632 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e36fd4df41c6493aa474b2ee1ff44509 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e36fd4df41c6493aa474b2ee1ff44509 2023-03-31 18:56:46,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ee4093a270c648cba1fff0cf834d6511 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ee4093a270c648cba1fff0cf834d6511 2023-03-31 18:56:46,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/0b4d88e895b843d098f475c93632c491 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/0b4d88e895b843d098f475c93632c491 2023-03-31 18:56:46,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/6fb70033454e4d77ad253b7b6d79d9cb to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/6fb70033454e4d77ad253b7b6d79d9cb 2023-03-31 18:56:46,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73f6b473ebd34e89b66b065a69ced439 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73f6b473ebd34e89b66b065a69ced439 2023-03-31 18:56:46,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/eb56197c414f499b93c496092c7067c8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/eb56197c414f499b93c496092c7067c8 2023-03-31 18:56:46,637 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9ff411089d434bd68a8677d942d32738 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9ff411089d434bd68a8677d942d32738 2023-03-31 18:56:46,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e05bf0a0ea914454b7e3a714639d88e1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/e05bf0a0ea914454b7e3a714639d88e1 2023-03-31 18:56:46,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2e34044873724ad79d7fcae000cf643d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2e34044873724ad79d7fcae000cf643d 2023-03-31 18:56:46,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73c5ef4e8be740f9bf929200d02581d4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/73c5ef4e8be740f9bf929200d02581d4 2023-03-31 18:56:46,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a6a06d81c12a49c4aa3088b238a41efd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a6a06d81c12a49c4aa3088b238a41efd 2023-03-31 18:56:46,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9456146bac7e4d3ebc350ad617442780 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9456146bac7e4d3ebc350ad617442780 2023-03-31 18:56:46,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/43aee40f4cf34d6388ae2d0d7c229159 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/43aee40f4cf34d6388ae2d0d7c229159 2023-03-31 18:56:46,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/d492fd3079064ef6959d791b33c02233 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/d492fd3079064ef6959d791b33c02233 2023-03-31 18:56:46,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2a32d880f2fa4a22a41208cfafe3e78e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/2a32d880f2fa4a22a41208cfafe3e78e 2023-03-31 18:56:46,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/92caf5deb6444d839c3d4df34318a318 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/92caf5deb6444d839c3d4df34318a318 2023-03-31 18:56:46,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/607dbd5229e748908c3ab73bcd1218a9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/607dbd5229e748908c3ab73bcd1218a9 2023-03-31 18:56:46,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/c37e93b1d4d24444a52d42774e5f8672 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/c37e93b1d4d24444a52d42774e5f8672 2023-03-31 18:56:46,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/8d25f0008b834586ba94717d621debda to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/8d25f0008b834586ba94717d621debda 2023-03-31 18:56:46,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9e14fc7cfd3d42229f9d6208abd22c8c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/9e14fc7cfd3d42229f9d6208abd22c8c 2023-03-31 18:56:46,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/67ff4b921b03467abaa9a6a0f64c6947 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/67ff4b921b03467abaa9a6a0f64c6947 2023-03-31 18:56:46,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/af4ae60919e24b8d9f88d77f1412e18e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/af4ae60919e24b8d9f88d77f1412e18e 2023-03-31 18:56:46,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ab513c6159ae495e9c1b41577f351ca9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ab513c6159ae495e9c1b41577f351ca9 2023-03-31 18:56:46,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ea17a3728fd2479d8bbb5f4bf7712abc to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/ea17a3728fd2479d8bbb5f4bf7712abc 2023-03-31 18:56:46,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1d0b1498a220480bab18c3b125bff379 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/1d0b1498a220480bab18c3b125bff379 2023-03-31 18:56:46,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/81607bd668854e7d940b17fb61746676 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/81607bd668854e7d940b17fb61746676 2023-03-31 18:56:46,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f20d0b1993bf4e90b28ec6cd2313294e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/f20d0b1993bf4e90b28ec6cd2313294e 2023-03-31 18:56:46,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/bb241f57fd764c68ab80d27f054500f6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/bb241f57fd764c68ab80d27f054500f6 2023-03-31 18:56:46,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a2eb87dade0843789775bf11338ab842 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/a2eb87dade0843789775bf11338ab842 2023-03-31 18:56:46,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c070c7bdc2314a8f81fdaa04c98dbaa2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2ee9677781ef4475b64d812daa1e3db0, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9e3570ff6a154ee9ab13f8b6f2c64992, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/60953f515e8b451682dc8c67725279ed, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/cb156128cd3c4bea808718aa8c63efb6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/632780cef648409eaa7b9ab0fe4e5300, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3110c06252a14e8aa9b68f26bd5060b7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3ec5ca73b3284d9eb23e48e03b2693bd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c6ff96acc3ab4838b57e2dba6abb879b, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d7067f38ba654fa690b4ec6200d16112, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9034b3ac7e6e49089b7b40822d004865, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/7cf4e3090cbf4beaba15bdc46b9a9bf1, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/fcb83efc22914babbc938e3874953edd, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d61671cd4ecb4d42b815caa3fa575150, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bba7259516eb46cc9198ee63135d6da9, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bbd18f62bc894274b6061e25c11c34e5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/20f10031814e4404ad1bb535dfd86f37, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9545295a115e4cebb4840f6b50f2db82, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/013bb6814cb149dc85e58579ed86866d, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8e10d8e674b343099813a6bd10c79d0c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d79719dc291a453786c67f4015df6fb2, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/34cfcff182144f649d140f0ea66df39c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/a987f28ad83f47f58006870fd68d09ad, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/701665bcef6b415f86604569d34eddc5, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/735487691bbc4fb1a414b9d9b14a8e5c, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/13ce14112d8543ed8536fe39bba80363, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/036c9cfc112b45aca53945f0235f7bc7, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/70ff5747993e4cd580938d7969a22ec6, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2428150d4f724ecf91a9d636eb2b6a03, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8cc3f722da974d218c7f69a6f4306c96, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/e3fc9a7ae0fa497d8d96049942e15fac, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/4a61a23553c1403fb9af01743778a469] to archive 2023-03-31 18:56:46,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 18:56:46,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c070c7bdc2314a8f81fdaa04c98dbaa2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c070c7bdc2314a8f81fdaa04c98dbaa2 2023-03-31 18:56:46,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2ee9677781ef4475b64d812daa1e3db0 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2ee9677781ef4475b64d812daa1e3db0 2023-03-31 18:56:46,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9e3570ff6a154ee9ab13f8b6f2c64992 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9e3570ff6a154ee9ab13f8b6f2c64992 2023-03-31 18:56:46,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/60953f515e8b451682dc8c67725279ed to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/60953f515e8b451682dc8c67725279ed 2023-03-31 18:56:46,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/cb156128cd3c4bea808718aa8c63efb6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/cb156128cd3c4bea808718aa8c63efb6 2023-03-31 18:56:46,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/632780cef648409eaa7b9ab0fe4e5300 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/632780cef648409eaa7b9ab0fe4e5300 2023-03-31 18:56:46,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3110c06252a14e8aa9b68f26bd5060b7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3110c06252a14e8aa9b68f26bd5060b7 2023-03-31 18:56:46,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3ec5ca73b3284d9eb23e48e03b2693bd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/3ec5ca73b3284d9eb23e48e03b2693bd 2023-03-31 18:56:46,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c6ff96acc3ab4838b57e2dba6abb879b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/c6ff96acc3ab4838b57e2dba6abb879b 2023-03-31 18:56:46,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d7067f38ba654fa690b4ec6200d16112 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d7067f38ba654fa690b4ec6200d16112 2023-03-31 18:56:46,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9034b3ac7e6e49089b7b40822d004865 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9034b3ac7e6e49089b7b40822d004865 2023-03-31 18:56:46,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/7cf4e3090cbf4beaba15bdc46b9a9bf1 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/7cf4e3090cbf4beaba15bdc46b9a9bf1 2023-03-31 18:56:46,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/fcb83efc22914babbc938e3874953edd to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/fcb83efc22914babbc938e3874953edd 2023-03-31 18:56:46,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d61671cd4ecb4d42b815caa3fa575150 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d61671cd4ecb4d42b815caa3fa575150 2023-03-31 18:56:46,673 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bba7259516eb46cc9198ee63135d6da9 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bba7259516eb46cc9198ee63135d6da9 2023-03-31 18:56:46,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bbd18f62bc894274b6061e25c11c34e5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/bbd18f62bc894274b6061e25c11c34e5 2023-03-31 18:56:46,675 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/20f10031814e4404ad1bb535dfd86f37 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/20f10031814e4404ad1bb535dfd86f37 2023-03-31 18:56:46,676 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9545295a115e4cebb4840f6b50f2db82 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/9545295a115e4cebb4840f6b50f2db82 2023-03-31 18:56:46,677 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/013bb6814cb149dc85e58579ed86866d to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/013bb6814cb149dc85e58579ed86866d 2023-03-31 18:56:46,678 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8e10d8e674b343099813a6bd10c79d0c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8e10d8e674b343099813a6bd10c79d0c 2023-03-31 18:56:46,679 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d79719dc291a453786c67f4015df6fb2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/d79719dc291a453786c67f4015df6fb2 2023-03-31 18:56:46,679 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/34cfcff182144f649d140f0ea66df39c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/34cfcff182144f649d140f0ea66df39c 2023-03-31 18:56:46,680 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/a987f28ad83f47f58006870fd68d09ad to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/a987f28ad83f47f58006870fd68d09ad 2023-03-31 18:56:46,681 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/701665bcef6b415f86604569d34eddc5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/701665bcef6b415f86604569d34eddc5 2023-03-31 18:56:46,682 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/735487691bbc4fb1a414b9d9b14a8e5c to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/735487691bbc4fb1a414b9d9b14a8e5c 2023-03-31 18:56:46,683 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/13ce14112d8543ed8536fe39bba80363 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/13ce14112d8543ed8536fe39bba80363 2023-03-31 18:56:46,684 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/036c9cfc112b45aca53945f0235f7bc7 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/036c9cfc112b45aca53945f0235f7bc7 2023-03-31 18:56:46,685 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/70ff5747993e4cd580938d7969a22ec6 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/70ff5747993e4cd580938d7969a22ec6 2023-03-31 18:56:46,686 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2428150d4f724ecf91a9d636eb2b6a03 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/2428150d4f724ecf91a9d636eb2b6a03 2023-03-31 18:56:46,687 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8cc3f722da974d218c7f69a6f4306c96 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/8cc3f722da974d218c7f69a6f4306c96 2023-03-31 18:56:46,688 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/e3fc9a7ae0fa497d8d96049942e15fac to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/e3fc9a7ae0fa497d8d96049942e15fac 2023-03-31 18:56:46,689 DEBUG [StoreCloser-TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/4a61a23553c1403fb9af01743778a469 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/4a61a23553c1403fb9af01743778a469 2023-03-31 18:56:46,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/recovered.edits/595.seqid, newMaxSeqId=595, maxSeqId=4 2023-03-31 18:56:46,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610. 2023-03-31 18:56:46,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f34b89af9bae1a93ef5b2acb275cb610: 2023-03-31 18:56:46,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:46,695 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=f34b89af9bae1a93ef5b2acb275cb610, regionState=CLOSED 2023-03-31 18:56:46,695 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680289006695"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680289006695"}]},"ts":"1680289006695"} 2023-03-31 18:56:46,696 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-31 18:56:46,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure f34b89af9bae1a93ef5b2acb275cb610, server=jenkins-hbase4.apache.org,34951,1680288816339 in 1.4960 sec 2023-03-31 18:56:46,698 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-31 18:56:46,698 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f34b89af9bae1a93ef5b2acb275cb610, UNASSIGN in 1.5000 sec 2023-03-31 18:56:46,698 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680289006698"}]},"ts":"1680289006698"} 2023-03-31 18:56:46,699 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 18:56:46,701 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 18:56:46,702 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5100 sec 2023-03-31 18:56:47,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 18:56:47,298 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-31 18:56:47,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 18:56:47,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:47,301 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:47,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-31 18:56:47,302 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:47,306 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:47,308 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C, FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/recovered.edits] 2023-03-31 18:56:47,310 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/885e7f69fde74339a1aff62069d384b4 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/885e7f69fde74339a1aff62069d384b4 2023-03-31 18:56:47,311 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/88a023b7ebff49269bbf8a5ca422ced8 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/88a023b7ebff49269bbf8a5ca422ced8 2023-03-31 18:56:47,312 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c7bc148d72ff4a83856bcc59fca04291 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/A/c7bc148d72ff4a83856bcc59fca04291 2023-03-31 18:56:47,314 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/030a6916dd414fcbb2ffe59bb2c2c300 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/030a6916dd414fcbb2ffe59bb2c2c300 2023-03-31 18:56:47,315 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/4cf88803a4784c788e86f3ca45566fea to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/4cf88803a4784c788e86f3ca45566fea 2023-03-31 18:56:47,316 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/7833a630fb034cdbb72dcac48bd00409 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/B/7833a630fb034cdbb72dcac48bd00409 2023-03-31 18:56:47,317 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/0c8a07ed7d4f43fa9d394ac51c708b65 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/0c8a07ed7d4f43fa9d394ac51c708b65 2023-03-31 18:56:47,318 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/38ffd8a07ab34d8ea74340f61e99a25a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/38ffd8a07ab34d8ea74340f61e99a25a 2023-03-31 18:56:47,319 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/681c538dc9fe4a9598a7111d79801575 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/C/681c538dc9fe4a9598a7111d79801575 2023-03-31 18:56:47,321 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/recovered.edits/595.seqid to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610/recovered.edits/595.seqid 2023-03-31 18:56:47,321 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/.tmp/data/default/TestAcidGuarantees/f34b89af9bae1a93ef5b2acb275cb610 2023-03-31 18:56:47,321 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 18:56:47,322 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 18:56:47,322 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-31 18:56:47,325 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033100ddb7876a9c446ab3a1b12b2e2f71fa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033100ddb7876a9c446ab3a1b12b2e2f71fa 2023-03-31 18:56:47,326 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331119e0eb14a18442aa543599fc5e87f7e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331119e0eb14a18442aa543599fc5e87f7e 2023-03-31 18:56:47,326 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331163dc1b159fa45a2b5d58cb14d7ec175 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331163dc1b159fa45a2b5d58cb14d7ec175 2023-03-31 18:56:47,327 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331206b28b546f744a39aa240301dc6ae35 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331206b28b546f744a39aa240301dc6ae35 2023-03-31 18:56:47,329 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033129c6579d2cf1483587a44e30dc2edfe5 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033129c6579d2cf1483587a44e30dc2edfe5 2023-03-31 18:56:47,330 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312b981a12300446cb81ee8c9f029d7fd3 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312b981a12300446cb81ee8c9f029d7fd3 2023-03-31 18:56:47,330 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312ba3e94992c54f79b4ccd90ccb785dfa to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312ba3e94992c54f79b4ccd90ccb785dfa 2023-03-31 18:56:47,331 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033135295f5805a24b088ad0f8b81a84f18a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033135295f5805a24b088ad0f8b81a84f18a 2023-03-31 18:56:47,332 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033147e57dcb32b444f79afa465a92320c99 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033147e57dcb32b444f79afa465a92320c99 2023-03-31 18:56:47,333 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315197a427b1cf46408d74f040ec297c85 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315197a427b1cf46408d74f040ec297c85 2023-03-31 18:56:47,334 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315dfa191a96a940c2a55d06948ee91c39 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315dfa191a96a940c2a55d06948ee91c39 2023-03-31 18:56:47,335 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315fe4b74cc0c84104b0064ca8b501f9ff to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315fe4b74cc0c84104b0064ca8b501f9ff 2023-03-31 18:56:47,335 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033166a52b03fc6147c09d4409161884d113 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033166a52b03fc6147c09d4409161884d113 2023-03-31 18:56:47,336 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331695a46b3408248bcbc2cb0c553c7e529 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331695a46b3408248bcbc2cb0c553c7e529 2023-03-31 18:56:47,337 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318321d657340f4dfaa31401a8dfb56805 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318321d657340f4dfaa31401a8dfb56805 2023-03-31 18:56:47,338 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033195ea9a6471f647e8ab516d4e568921e2 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033195ea9a6471f647e8ab516d4e568921e2 2023-03-31 18:56:47,339 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331aca054d543a743b6a1d98c2133396b06 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331aca054d543a743b6a1d98c2133396b06 2023-03-31 18:56:47,339 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b0e804afc7f44eb2894eaf4b8b501a20 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b0e804afc7f44eb2894eaf4b8b501a20 2023-03-31 18:56:47,340 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b5060d06cf784d3ca12dafc96fd3d885 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b5060d06cf784d3ca12dafc96fd3d885 2023-03-31 18:56:47,341 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331be574baee4714e5d8d8516fbaad1897b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331be574baee4714e5d8d8516fbaad1897b 2023-03-31 18:56:47,342 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c4f2a84cc83e4641887ef3712037629f to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c4f2a84cc83e4641887ef3712037629f 2023-03-31 18:56:47,342 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c4fb014554a04ae9ac9c1245e0d66083 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c4fb014554a04ae9ac9c1245e0d66083 2023-03-31 18:56:47,343 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ce3cb40c3c6f4b7aac183c46ffc0b2ff to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ce3cb40c3c6f4b7aac183c46ffc0b2ff 2023-03-31 18:56:47,344 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d0960788d9544d3391f7469f2af8fd8e to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d0960788d9544d3391f7469f2af8fd8e 2023-03-31 18:56:47,345 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d11a399e9ab34a1d96f9c2c74762804a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d11a399e9ab34a1d96f9c2c74762804a 2023-03-31 18:56:47,345 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ecf8092d4c8546979e36ace133521f99 to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ecf8092d4c8546979e36ace133521f99 2023-03-31 18:56:47,346 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f52f1899f97d458aa5680006574c0c5a to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f52f1899f97d458aa5680006574c0c5a 2023-03-31 18:56:47,347 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331fd0c665423444b71aa7f7bb0a253ba8b to hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331fd0c665423444b71aa7f7bb0a253ba8b 2023-03-31 18:56:47,347 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 18:56:47,349 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:47,350 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 18:56:47,352 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 18:56:47,352 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:47,352 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 18:56:47,352 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680289007352"}]},"ts":"9223372036854775807"} 2023-03-31 18:56:47,354 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 18:56:47,354 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f34b89af9bae1a93ef5b2acb275cb610, NAME => 'TestAcidGuarantees,,1680288978280.f34b89af9bae1a93ef5b2acb275cb610.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 18:56:47,354 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 18:56:47,354 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680289007354"}]},"ts":"9223372036854775807"} 2023-03-31 18:56:47,355 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 18:56:47,356 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 18:56:47,357 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-03-31 18:56:47,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35737] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-31 18:56:47,403 INFO [Listener at localhost/43327] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-31 18:56:47,414 INFO [Listener at localhost/43327] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227 (was 228), OpenFileDescriptor=565 (was 591), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=383 (was 353) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3838 (was 3761) - AvailableMemoryMB LEAK? - 2023-03-31 18:56:47,414 INFO [Listener at localhost/43327] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-31 18:56:47,414 INFO [Listener at localhost/43327] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 18:56:47,415 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e324982 to 127.0.0.1:63858 2023-03-31 18:56:47,415 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:47,415 DEBUG [Listener at localhost/43327] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-31 18:56:47,415 DEBUG [Listener at localhost/43327] util.JVMClusterUtil(257): Found active master hash=2012977623, stopped=false 2023-03-31 18:56:47,415 INFO [Listener at localhost/43327] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,35737,1680288815092 2023-03-31 18:56:47,416 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-31 18:56:47,417 INFO [Listener at localhost/43327] procedure2.ProcedureExecutor(629): Stopping 2023-03-31 18:56:47,417 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-31 18:56:47,417 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 18:56:47,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-31 18:56:47,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-31 18:56:47,420 DEBUG [Listener at localhost/43327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ffc8fa3 to 127.0.0.1:63858 2023-03-31 18:56:47,420 DEBUG [Listener at localhost/43327] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:47,420 INFO [Listener at localhost/43327] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34951,1680288816339' ***** 2023-03-31 18:56:47,420 INFO [Listener at localhost/43327] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-31 18:56:47,420 INFO [RS:0;jenkins-hbase4:34951] regionserver.HeapMemoryManager(220): Stopping 2023-03-31 18:56:47,421 INFO [RS:0;jenkins-hbase4:34951] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-31 18:56:47,421 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-31 18:56:47,421 INFO [RS:0;jenkins-hbase4:34951] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-31 18:56:47,421 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(3304): Received CLOSE for 2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:56:47,422 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:47,422 DEBUG [RS:0;jenkins-hbase4:34951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b3fb4aa to 127.0.0.1:63858 2023-03-31 18:56:47,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2599f97ef8c905b3cf5450aad14b5536, disabling compactions & flushes 2023-03-31 18:56:47,422 DEBUG [RS:0;jenkins-hbase4:34951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:47,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:56:47,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:56:47,422 INFO [RS:0;jenkins-hbase4:34951] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-31 18:56:47,422 INFO [RS:0;jenkins-hbase4:34951] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-31 18:56:47,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. after waiting 0 ms 2023-03-31 18:56:47,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:56:47,422 INFO [RS:0;jenkins-hbase4:34951] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-31 18:56:47,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2599f97ef8c905b3cf5450aad14b5536 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-31 18:56:47,423 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-31 18:56:47,423 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-31 18:56:47,423 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 2599f97ef8c905b3cf5450aad14b5536=hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536.} 2023-03-31 18:56:47,423 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-31 18:56:47,424 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-31 18:56:47,424 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-31 18:56:47,424 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-31 18:56:47,424 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-31 18:56:47,424 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-31 18:56:47,425 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1505): Waiting on 1588230740, 2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:56:47,518 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-31 18:56:47,625 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1505): Waiting on 1588230740, 2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:56:47,682 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-31 18:56:47,682 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-31 18:56:47,826 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1505): Waiting on 1588230740, 2599f97ef8c905b3cf5450aad14b5536 2023-03-31 18:56:47,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536/.tmp/info/49bfe0ac6b0047169b50b03772b51e6b 2023-03-31 18:56:47,832 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/.tmp/info/432f6b8ba32c4d8f90d5f245c4d9c67d 2023-03-31 18:56:47,836 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 432f6b8ba32c4d8f90d5f245c4d9c67d 2023-03-31 18:56:47,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536/.tmp/info/49bfe0ac6b0047169b50b03772b51e6b as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536/info/49bfe0ac6b0047169b50b03772b51e6b 2023-03-31 18:56:47,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536/info/49bfe0ac6b0047169b50b03772b51e6b, entries=2, sequenceid=6, filesize=4.8 K 2023-03-31 18:56:47,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 2599f97ef8c905b3cf5450aad14b5536 in 417ms, sequenceid=6, compaction requested=false 2023-03-31 18:56:47,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/namespace/2599f97ef8c905b3cf5450aad14b5536/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-31 18:56:47,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:56:47,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2599f97ef8c905b3cf5450aad14b5536: 2023-03-31 18:56:47,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680288818186.2599f97ef8c905b3cf5450aad14b5536. 2023-03-31 18:56:48,026 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-31 18:56:48,226 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-31 18:56:48,245 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/.tmp/rep_barrier/4bbb0eb6bf214b378d22e2a033365750 2023-03-31 18:56:48,248 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4bbb0eb6bf214b378d22e2a033365750 2023-03-31 18:56:48,426 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-03-31 18:56:48,427 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-03-31 18:56:48,427 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-31 18:56:48,627 DEBUG [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-31 18:56:48,656 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/.tmp/table/50ddf282e8b64a9db7778d87d183c951 2023-03-31 18:56:48,659 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 50ddf282e8b64a9db7778d87d183c951 2023-03-31 18:56:48,659 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/.tmp/info/432f6b8ba32c4d8f90d5f245c4d9c67d as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/info/432f6b8ba32c4d8f90d5f245c4d9c67d 2023-03-31 18:56:48,662 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 432f6b8ba32c4d8f90d5f245c4d9c67d 2023-03-31 18:56:48,662 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/info/432f6b8ba32c4d8f90d5f245c4d9c67d, entries=22, sequenceid=93, filesize=7.1 K 2023-03-31 18:56:48,664 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/.tmp/rep_barrier/4bbb0eb6bf214b378d22e2a033365750 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/rep_barrier/4bbb0eb6bf214b378d22e2a033365750 2023-03-31 18:56:48,667 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4bbb0eb6bf214b378d22e2a033365750 2023-03-31 18:56:48,667 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/rep_barrier/4bbb0eb6bf214b378d22e2a033365750, entries=6, sequenceid=93, filesize=5.5 K 2023-03-31 18:56:48,667 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/.tmp/table/50ddf282e8b64a9db7778d87d183c951 as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/table/50ddf282e8b64a9db7778d87d183c951 2023-03-31 18:56:48,670 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 50ddf282e8b64a9db7778d87d183c951 2023-03-31 18:56:48,670 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/table/50ddf282e8b64a9db7778d87d183c951, entries=9, sequenceid=93, filesize=5.5 K 2023-03-31 18:56:48,670 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1246ms, sequenceid=93, compaction requested=false 2023-03-31 18:56:48,674 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-31 18:56:48,675 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-31 18:56:48,676 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-31 18:56:48,676 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-31 18:56:48,676 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-31 18:56:48,827 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34951,1680288816339; all regions closed. 2023-03-31 18:56:48,835 DEBUG [RS:0;jenkins-hbase4:34951] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/oldWALs 2023-03-31 18:56:48,835 INFO [RS:0;jenkins-hbase4:34951] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34951%2C1680288816339.meta:.meta(num 1680288817935) 2023-03-31 18:56:48,839 DEBUG [RS:0;jenkins-hbase4:34951] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/oldWALs 2023-03-31 18:56:48,839 INFO [RS:0;jenkins-hbase4:34951] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34951%2C1680288816339:(num 1680288817840) 2023-03-31 18:56:48,839 DEBUG [RS:0;jenkins-hbase4:34951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:48,839 INFO [RS:0;jenkins-hbase4:34951] regionserver.LeaseManager(133): Closed leases 2023-03-31 18:56:48,839 INFO [RS:0;jenkins-hbase4:34951] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-31 18:56:48,840 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-31 18:56:48,841 INFO [RS:0;jenkins-hbase4:34951] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34951 2023-03-31 18:56:48,846 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34951,1680288816339 2023-03-31 18:56:48,846 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-31 18:56:48,846 ERROR [Listener at localhost/43327-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@50c61bf7 rejected from java.util.concurrent.ThreadPoolExecutor@54f98537[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2635] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 18:56:48,847 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-31 18:56:48,847 ERROR [Listener at localhost/43327-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@40d9a662 rejected from java.util.concurrent.ThreadPoolExecutor@54f98537[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2635] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 18:56:48,852 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34951,1680288816339] 2023-03-31 18:56:48,853 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34951,1680288816339; numProcessing=1 2023-03-31 18:56:48,854 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34951,1680288816339 already deleted, retry=false 2023-03-31 18:56:48,854 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34951,1680288816339 expired; onlineServers=0 2023-03-31 18:56:48,854 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35737,1680288815092' ***** 2023-03-31 18:56:48,854 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-31 18:56:48,854 DEBUG [M:0;jenkins-hbase4:35737] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36c693ec, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-31 18:56:48,854 INFO [M:0;jenkins-hbase4:35737] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35737,1680288815092 2023-03-31 18:56:48,855 INFO [M:0;jenkins-hbase4:35737] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35737,1680288815092; all regions closed. 2023-03-31 18:56:48,855 DEBUG [M:0;jenkins-hbase4:35737] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 18:56:48,855 DEBUG [M:0;jenkins-hbase4:35737] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-31 18:56:48,855 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-31 18:56:48,855 DEBUG [M:0;jenkins-hbase4:35737] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-31 18:56:48,855 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680288817456] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680288817456,5,FailOnTimeoutGroup] 2023-03-31 18:56:48,855 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680288817456] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680288817456,5,FailOnTimeoutGroup] 2023-03-31 18:56:48,855 INFO [M:0;jenkins-hbase4:35737] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-31 18:56:48,855 INFO [M:0;jenkins-hbase4:35737] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-31 18:56:48,855 INFO [M:0;jenkins-hbase4:35737] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-31 18:56:48,856 DEBUG [M:0;jenkins-hbase4:35737] master.HMaster(1512): Stopping service threads 2023-03-31 18:56:48,856 INFO [M:0;jenkins-hbase4:35737] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-31 18:56:48,856 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-31 18:56:48,856 ERROR [M:0;jenkins-hbase4:35737] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-03-31 18:56:48,856 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase Thread[IPC Client (1760173019) connection to localhost/127.0.0.1:44569 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-31 18:56:48,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-31 18:56:48,857 INFO [M:0;jenkins-hbase4:35737] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-31 18:56:48,857 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-31 18:56:48,857 DEBUG [M:0;jenkins-hbase4:35737] zookeeper.ZKUtil(398): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-31 18:56:48,857 WARN [M:0;jenkins-hbase4:35737] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-31 18:56:48,857 INFO [M:0;jenkins-hbase4:35737] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-31 18:56:48,857 INFO [M:0;jenkins-hbase4:35737] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-31 18:56:48,858 DEBUG [M:0;jenkins-hbase4:35737] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-31 18:56:48,858 INFO [M:0;jenkins-hbase4:35737] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 18:56:48,858 DEBUG [M:0;jenkins-hbase4:35737] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 18:56:48,858 DEBUG [M:0;jenkins-hbase4:35737] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-31 18:56:48,858 DEBUG [M:0;jenkins-hbase4:35737] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 18:56:48,858 INFO [M:0;jenkins-hbase4:35737] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.82 KB heapSize=393.59 KB 2023-03-31 18:56:48,953 INFO [RS:0;jenkins-hbase4:34951] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34951,1680288816339; zookeeper connection closed. 2023-03-31 18:56:48,953 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-31 18:56:48,953 ERROR [Listener at localhost/43327-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@38e8ed23 rejected from java.util.concurrent.ThreadPoolExecutor@54f98537[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2635] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 18:56:48,954 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): regionserver:34951-0x1013bec7e730001, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-31 18:56:48,954 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7db7b8ea] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7db7b8ea 2023-03-31 18:56:48,954 ERROR [Listener at localhost/43327-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7c33bc6d rejected from java.util.concurrent.ThreadPoolExecutor@54f98537[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2635] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 18:56:48,955 INFO [Listener at localhost/43327] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-31 18:56:49,266 INFO [M:0;jenkins-hbase4:35737] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.82 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/922a0fe1c0a446eda385310a078d241f 2023-03-31 18:56:49,269 INFO [M:0;jenkins-hbase4:35737] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 922a0fe1c0a446eda385310a078d241f 2023-03-31 18:56:49,270 DEBUG [M:0;jenkins-hbase4:35737] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/922a0fe1c0a446eda385310a078d241f as hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/922a0fe1c0a446eda385310a078d241f 2023-03-31 18:56:49,272 INFO [M:0;jenkins-hbase4:35737] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 922a0fe1c0a446eda385310a078d241f 2023-03-31 18:56:49,272 INFO [M:0;jenkins-hbase4:35737] regionserver.HStore(1080): Added hdfs://localhost:44569/user/jenkins/test-data/5ee0f866-1194-d9e0-d2aa-60379be6a6a9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/922a0fe1c0a446eda385310a078d241f, entries=65, sequenceid=578, filesize=24.6 K 2023-03-31 18:56:49,273 INFO [M:0;jenkins-hbase4:35737] regionserver.HRegion(2948): Finished flush of dataSize ~344.82 KB/353097, heapSize ~393.58 KB/403024, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 415ms, sequenceid=578, compaction requested=false 2023-03-31 18:56:49,274 INFO [M:0;jenkins-hbase4:35737] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 18:56:49,274 DEBUG [M:0;jenkins-hbase4:35737] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-31 18:56:49,276 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-31 18:56:49,276 INFO [M:0;jenkins-hbase4:35737] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-31 18:56:49,277 INFO [M:0;jenkins-hbase4:35737] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35737 2023-03-31 18:56:49,279 DEBUG [M:0;jenkins-hbase4:35737] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,35737,1680288815092 already deleted, retry=false 2023-03-31 18:56:49,382 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-31 18:56:49,382 INFO [M:0;jenkins-hbase4:35737] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35737,1680288815092; zookeeper connection closed. 2023-03-31 18:56:49,382 ERROR [Listener at localhost/43327-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5ac865ce rejected from java.util.concurrent.ThreadPoolExecutor@2c4d1e16[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4398] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 18:56:49,382 DEBUG [Listener at localhost/43327-EventThread] zookeeper.ZKWatcher(600): master:35737-0x1013bec7e730000, quorum=127.0.0.1:63858, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-31 18:56:49,382 ERROR [Listener at localhost/43327-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6a4f439d rejected from java.util.concurrent.ThreadPoolExecutor@2c4d1e16[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4398] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 18:56:49,384 WARN [Listener at localhost/43327] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-31 18:56:49,387 INFO [Listener at localhost/43327] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-31 18:56:49,495 WARN [BP-2799142-172.31.14.131-1680288811800 heartbeating to localhost/127.0.0.1:44569] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-31 18:56:49,495 WARN [BP-2799142-172.31.14.131-1680288811800 heartbeating to localhost/127.0.0.1:44569] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-2799142-172.31.14.131-1680288811800 (Datanode Uuid 2ae9146c-fb95-48c5-9b36-7f399d9cd4f0) service to localhost/127.0.0.1:44569 2023-03-31 18:56:49,499 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/cluster_a78361e5-c460-aaf7-5d30-b8a2ca8e7b70/dfs/data/data1/current/BP-2799142-172.31.14.131-1680288811800] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-31 18:56:49,500 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ac344c-0bbe-f2c2-9887-f46fb52b7a82/cluster_a78361e5-c460-aaf7-5d30-b8a2ca8e7b70/dfs/data/data2/current/BP-2799142-172.31.14.131-1680288811800] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-31 18:56:49,532 INFO [Listener at localhost/43327] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-31 18:56:49,702 INFO [Listener at localhost/43327] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-31 18:56:49,873 INFO [Listener at localhost/43327] hbase.HBaseTestingUtility(1293): Minicluster is down